| Message ID | 20240415083019.1867284-1-changqing.li@windriver.com |
|---|---|
| State | New |
| Headers | show |
| Series | base.bbclass: make do_fetch depends on PREMIRRORS MIRRORS | expand |
How does this work when sharing sstate between builders with different PREMIRRORS/MIRRORS? Doesn't this cause all signatures to be different as do_unpack does depend on do_fetch? On Mon, Apr 15, 2024 at 10:30 AM Changqing Li via lists.openembedded.org <changqing.li=eng.windriver.com@lists.openembedded.org> wrote: > > From: Changqing Li <changqing.li@windriver.com> > > When there are tarballs in local mirror file://, do_fetch will > create symlink under DL_DIR, eg: > pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz > after user update path from localmirrorA to localmirrorB, because > do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not > be rerun, so the symlink will not update to: > pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz > then, do_upack will failed since the tarball cannot be found. > > Signed-off-by: Changqing Li <changqing.li@windriver.com> > --- > meta/classes-global/base.bbclass | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass > index 0999b42daa..5094a1540e 100644 > --- a/meta/classes-global/base.bbclass > +++ b/meta/classes-global/base.bbclass > @@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}" > do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}" > do_fetch[prefuncs] += "fetcher_hashes_dummyfunc" > do_fetch[network] = "1" > +do_fetch[vardeps] += "PREMIRRORS MIRRORS" > python base_do_fetch() { > > src_uri = (d.getVar('SRC_URI') or "").split() > -- > 2.25.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#198222): https://lists.openembedded.org/g/openembedded-core/message/198222 > Mute This Topic: https://lists.openembedded.org/mt/105532159/3617156 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [martin.jansa@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Mon, 2024-04-15 at 16:30 +0800, Changqing Li via lists.openembedded.org wrote: > From: Changqing Li <changqing.li@windriver.com> > > When there are tarballs in local mirror file://, do_fetch will > create symlink under DL_DIR, eg: > pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz > after user update path from localmirrorA to localmirrorB, because > do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not > be rerun, so the symlink will not update to: > pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz > then, do_upack will failed since the tarball cannot be found. > > Signed-off-by: Changqing Li <changqing.li@windriver.com> > --- > meta/classes-global/base.bbclass | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass > index 0999b42daa..5094a1540e 100644 > --- a/meta/classes-global/base.bbclass > +++ b/meta/classes-global/base.bbclass > @@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}" > do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}" > do_fetch[prefuncs] += "fetcher_hashes_dummyfunc" > do_fetch[network] = "1" > +do_fetch[vardeps] += "PREMIRRORS MIRRORS" > python base_do_fetch() { > > src_uri = (d.getVar('SRC_URI') or "").split() > Changing PREMIRRORS or MIRRORS should not change the output of the task, ever, so do_fetch shouldn't depend upon them. Cheers, Richard
On 4/15/24 18:05, Richard Purdie wrote: > CAUTION: This email comes from a non Wind River email account! > Do not click links or open attachments unless you recognize the sender and know the content is safe. > > On Mon, 2024-04-15 at 16:30 +0800, Changqing Li via lists.openembedded.org wrote: >> From: Changqing Li <changqing.li@windriver.com> >> >> When there are tarballs in local mirror file://, do_fetch will >> create symlink under DL_DIR, eg: >> pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz >> after user update path from localmirrorA to localmirrorB, because >> do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not >> be rerun, so the symlink will not update to: >> pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz >> then, do_upack will failed since the tarball cannot be found. >> >> Signed-off-by: Changqing Li <changqing.li@windriver.com> >> --- >> meta/classes-global/base.bbclass | 1 + >> 1 file changed, 1 insertion(+) >> >> diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass >> index 0999b42daa..5094a1540e 100644 >> --- a/meta/classes-global/base.bbclass >> +++ b/meta/classes-global/base.bbclass >> @@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}" >> do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}" >> do_fetch[prefuncs] += "fetcher_hashes_dummyfunc" >> do_fetch[network] = "1" >> +do_fetch[vardeps] += "PREMIRRORS MIRRORS" >> python base_do_fetch() { >> >> src_uri = (d.getVar('SRC_URI') or "").split() >> > Changing PREMIRRORS or MIRRORS should not change the output of the > task, ever, so do_fetch shouldn't depend upon them. The failure case I met is like this: I have a project with PREMIRRORS setting, and the tarballs are under it. PREMIRRORS:append = "file:///project/path/to/localmirrorA" after first bitbake pseudo-native, there will a symlink under DL_DIR: pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz Then, Since all the tarballs are moved to /project/path/to/localmirrorB I changed PREMIRRORS:append = "file:///project/path/to/localmirrorB" Then bitbake pseudo-native again, do_fetch will not rerun since no signature change. do_unpack rerun since one dependency change, it will try to unpack ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz, but since do_fetch not rerun, ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz still point to localmirrorA, but localmirrorA not exist since localmirrorB replace it. Then will cause do_unpck failure: NOTE: Unpacking /build/downloads/pseudo-prebuilt-2.33.tar.xz to /build/tmp-glibc/work/x86_64-linux/pseudo-native/1.9.0+git xz: /build/downloads/pseudo-prebuilt-2.33.tar.xz: No such file or directory So this is why I want to add this depends, and want to make do_fetch rerun and resymlink to /project/path/to/localmirrorB Regards Changqing > > Cheers, > > Richard >
I'm not sure why files in DL_DIR would symlink to the local (pre)mirrors instead of making an independent copy? Mirrors can indeed move or disappear altogether. Can you look into where the symlinking happens, and whether that can be modified or configured differently (maybe also the git history for that code)? Alex On Tue, 16 Apr 2024 at 03:48, Changqing Li via lists.openembedded.org <changqing.li=eng.windriver.com@lists.openembedded.org> wrote: > > > On 4/15/24 18:05, Richard Purdie wrote: > > CAUTION: This email comes from a non Wind River email account! > > Do not click links or open attachments unless you recognize the sender and know the content is safe. > > > > On Mon, 2024-04-15 at 16:30 +0800, Changqing Li via lists.openembedded.org wrote: > >> From: Changqing Li <changqing.li@windriver.com> > >> > >> When there are tarballs in local mirror file://, do_fetch will > >> create symlink under DL_DIR, eg: > >> pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz > >> after user update path from localmirrorA to localmirrorB, because > >> do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not > >> be rerun, so the symlink will not update to: > >> pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz > >> then, do_upack will failed since the tarball cannot be found. > >> > >> Signed-off-by: Changqing Li <changqing.li@windriver.com> > >> --- > >> meta/classes-global/base.bbclass | 1 + > >> 1 file changed, 1 insertion(+) > >> > >> diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass > >> index 0999b42daa..5094a1540e 100644 > >> --- a/meta/classes-global/base.bbclass > >> +++ b/meta/classes-global/base.bbclass > >> @@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}" > >> do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}" > >> do_fetch[prefuncs] += "fetcher_hashes_dummyfunc" > >> do_fetch[network] = "1" > >> +do_fetch[vardeps] += "PREMIRRORS MIRRORS" > >> python base_do_fetch() { > >> > >> src_uri = (d.getVar('SRC_URI') or "").split() > >> > > Changing PREMIRRORS or MIRRORS should not change the output of the > > task, ever, so do_fetch shouldn't depend upon them. > > The failure case I met is like this: > > I have a project with PREMIRRORS setting, and the tarballs are under it. > > PREMIRRORS:append = "file:///project/path/to/localmirrorA" > > after first bitbake pseudo-native, there will a symlink under DL_DIR: > > pseudo-prebuilt-2.33.tar.xz -> > /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz > > Then, Since all the tarballs are moved to /project/path/to/localmirrorB > > I changed PREMIRRORS:append = "file:///project/path/to/localmirrorB" > > Then bitbake pseudo-native again, do_fetch will not rerun since no > signature change. > > do_unpack rerun since one dependency change, it will try to unpack > ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz, > > but since do_fetch not rerun, ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz > still point to localmirrorA, but localmirrorA not exist > > since localmirrorB replace it. Then will cause do_unpck failure: > > NOTE: Unpacking /build/downloads/pseudo-prebuilt-2.33.tar.xz to > /build/tmp-glibc/work/x86_64-linux/pseudo-native/1.9.0+git xz: > /build/downloads/pseudo-prebuilt-2.33.tar.xz: No such file or directory > > So this is why I want to add this depends, and want to make do_fetch > rerun and resymlink to /project/path/to/localmirrorB > > Regards > > Changqing > > > > > Cheers, > > > > Richard > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#198246): https://lists.openembedded.org/g/openembedded-core/message/198246 > Mute This Topic: https://lists.openembedded.org/mt/105532159/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On 4/16/24 15:30, Alexander Kanavin wrote: > CAUTION: This email comes from a non Wind River email account! > Do not click links or open attachments unless you recognize the sender and know the content is safe. > > I'm not sure why files in DL_DIR would symlink to the local > (pre)mirrors instead of making an independent copy? Mirrors can indeed > move or disappear altogether. > > Can you look into where the symlinking happens, and whether that can > be modified or configured differently (maybe also the git history for > that code)? Current code is here: https://git.openembedded.org/bitbake/tree/lib/bb/fetch2/__init__.py#n1096 It is created during handle the (pre)mirrors in do_fetch. And the history long time ago: https://git.openembedded.org/bitbake/commit/?id=83f9445081595630488a5d3e170d282a13836c98 About if we can change to independent copy, I need to do more research, welcome to provide me some background and notes. Regards Changqing > > Alex > > On Tue, 16 Apr 2024 at 03:48, Changqing Li via lists.openembedded.org > <changqing.li=eng.windriver.com@lists.openembedded.org> wrote: >> >> On 4/15/24 18:05, Richard Purdie wrote: >>> CAUTION: This email comes from a non Wind River email account! >>> Do not click links or open attachments unless you recognize the sender and know the content is safe. >>> >>> On Mon, 2024-04-15 at 16:30 +0800, Changqing Li via lists.openembedded.org wrote: >>>> From: Changqing Li<changqing.li@windriver.com> >>>> >>>> When there are tarballs in local mirrorfile://, do_fetch will >>>> create symlink under DL_DIR, eg: >>>> pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz >>>> after user update path from localmirrorA to localmirrorB, because >>>> do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not >>>> be rerun, so the symlink will not update to: >>>> pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz >>>> then, do_upack will failed since the tarball cannot be found. >>>> >>>> Signed-off-by: Changqing Li<changqing.li@windriver.com> >>>> --- >>>> meta/classes-global/base.bbclass | 1 + >>>> 1 file changed, 1 insertion(+) >>>> >>>> diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass >>>> index 0999b42daa..5094a1540e 100644 >>>> --- a/meta/classes-global/base.bbclass >>>> +++ b/meta/classes-global/base.bbclass >>>> @@ -139,6 +139,7 @@ do_fetch[file-checksums] ="${@bb.fetch.get_checksum_file_list(d)}" >>>> do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}" >>>> do_fetch[prefuncs] += "fetcher_hashes_dummyfunc" >>>> do_fetch[network] = "1" >>>> +do_fetch[vardeps] += "PREMIRRORS MIRRORS" >>>> python base_do_fetch() { >>>> >>>> src_uri = (d.getVar('SRC_URI') or "").split() >>>> >>> Changing PREMIRRORS or MIRRORS should not change the output of the >>> task, ever, so do_fetch shouldn't depend upon them. >> The failure case I met is like this: >> >> I have a project with PREMIRRORS setting, and the tarballs are under it. >> >> PREMIRRORS:append ="file:///project/path/to/localmirrorA" >> >> after first bitbake pseudo-native, there will a symlink under DL_DIR: >> >> pseudo-prebuilt-2.33.tar.xz -> >> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz >> >> Then, Since all the tarballs are moved to /project/path/to/localmirrorB >> >> I changed PREMIRRORS:append ="file:///project/path/to/localmirrorB" >> >> Then bitbake pseudo-native again, do_fetch will not rerun since no >> signature change. >> >> do_unpack rerun since one dependency change, it will try to unpack >> ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz, >> >> but since do_fetch not rerun, ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz >> still point to localmirrorA, but localmirrorA not exist >> >> since localmirrorB replace it. Then will cause do_unpck failure: >> >> NOTE: Unpacking /build/downloads/pseudo-prebuilt-2.33.tar.xz to >> /build/tmp-glibc/work/x86_64-linux/pseudo-native/1.9.0+git xz: >> /build/downloads/pseudo-prebuilt-2.33.tar.xz: No such file or directory >> >> So this is why I want to add this depends, and want to make do_fetch >> rerun and resymlink to /project/path/to/localmirrorB >> >> Regards >> >> Changqing >> >>> Cheers, >>> >>> Richard >>> >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> View/Reply Online (#198246):https://lists.openembedded.org/g/openembedded-core/message/198246 >> Mute This Topic:https://lists.openembedded.org/mt/105532159/1686489 >> Group Owner:openembedded-core+owner@lists.openembedded.org >> Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] >> -=-=-=-=-=-=-=-=-=-=-=- >>
On Tue, 2024-04-16 at 09:48 +0800, Changqing Li wrote: > On 4/15/24 18:05, Richard Purdie wrote: > > CAUTION: This email comes from a non Wind River email account! > > Do not click links or open attachments unless you recognize the > > sender and know the content is safe. > > > > On Mon, 2024-04-15 at 16:30 +0800, Changqing Li via > > lists.openembedded.org wrote: > > > From: Changqing Li <changqing.li@windriver.com> > > > > > > When there are tarballs in local mirror file://, do_fetch will > > > create symlink under DL_DIR, eg: > > > pseudo-prebuilt-2.33.tar.xz -> > > > /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz > > > after user update path from localmirrorA to localmirrorB, because > > > do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not > > > be rerun, so the symlink will not update to: > > > pseudo-prebuilt-2.33.tar.xz -> > > > /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz > > > then, do_upack will failed since the tarball cannot be found. > > > > > > Signed-off-by: Changqing Li <changqing.li@windriver.com> > > > --- > > > meta/classes-global/base.bbclass | 1 + > > > 1 file changed, 1 insertion(+) > > > > > > diff --git a/meta/classes-global/base.bbclass b/meta/classes- > > > global/base.bbclass > > > index 0999b42daa..5094a1540e 100644 > > > --- a/meta/classes-global/base.bbclass > > > +++ b/meta/classes-global/base.bbclass > > > @@ -139,6 +139,7 @@ do_fetch[file-checksums] = > > > "${@bb.fetch.get_checksum_file_list(d)}" > > > do_fetch[file-checksums] += " > > > ${@get_lic_checksum_file_list(d)}" > > > do_fetch[prefuncs] += "fetcher_hashes_dummyfunc" > > > do_fetch[network] = "1" > > > +do_fetch[vardeps] += "PREMIRRORS MIRRORS" > > > python base_do_fetch() { > > > > > > src_uri = (d.getVar('SRC_URI') or "").split() > > > > > Changing PREMIRRORS or MIRRORS should not change the output of the > > task, ever, so do_fetch shouldn't depend upon them. > > The failure case I met is like this: > > I have a project with PREMIRRORS setting, and the tarballs are under > it. > > PREMIRRORS:append = "file:///project/path/to/localmirrorA" > > after first bitbake pseudo-native, there will a symlink under DL_DIR: > > pseudo-prebuilt-2.33.tar.xz -> > /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz > > Then, Since all the tarballs are moved to > /project/path/to/localmirrorB > > I changed PREMIRRORS:append = "file:///project/path/to/localmirrorB" > > Then bitbake pseudo-native again, do_fetch will not rerun since no > signature change. > > do_unpack rerun since one dependency change, it will try to unpack > ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz, > > but since do_fetch not rerun, ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz > still point to localmirrorA, but localmirrorA not exist > > since localmirrorB replace it. Then will cause do_unpck failure: > > NOTE: Unpacking /build/downloads/pseudo-prebuilt-2.33.tar.xz to > /build/tmp-glibc/work/x86_64-linux/pseudo-native/1.9.0+git xz: > /build/downloads/pseudo-prebuilt-2.33.tar.xz: No such file or > directory > > So this is why I want to add this depends, and want to make do_fetch > rerun and resymlink to /project/path/to/localmirrorB The question is if the fetch reruns, does it correct the path? do_fetch should definitely not depend on PREMIRRORS or MIRRORS as those can be set by local users for local setups. The idea is that result would be the same regardless, else the fetch checksums (or git revision) would not match. I will therefore not take any patch changing that. Also, despite what Alex said, the fetcher does only make symlinks for local files, it does not copy every file into DL_DIR. It does this for efficiency, both time and disk usage. If you start moving files around locally, you're expected to fix up the links too. I believe that if you manually run the fetch task, it will update correctly so the only issue is it not automatically triggering but I don't believe it should trigger. So whilst I can see how you have an unfortunate build failure, it was of your own making by moving mirror directories around. Cheers, Richard
diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass index 0999b42daa..5094a1540e 100644 --- a/meta/classes-global/base.bbclass +++ b/meta/classes-global/base.bbclass @@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}" do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}" do_fetch[prefuncs] += "fetcher_hashes_dummyfunc" do_fetch[network] = "1" +do_fetch[vardeps] += "PREMIRRORS MIRRORS" python base_do_fetch() { src_uri = (d.getVar('SRC_URI') or "").split()