| Message ID | 20241014102817.40298-1-mikko.rapeli@linaro.org |
|---|---|
| State | New |
| Headers | show |
| Series | ovmf-native: remove .pyc files from install | expand |
This doesn't quite make sense. Why is the problem specific to .pyc files and not other files ovmf installs? Alex On Mon, 14 Oct 2024 at 12:28, Mikko Rapeli via lists.openembedded.org <mikko.rapeli=linaro.org@lists.openembedded.org> wrote: > > They break builds which share sstate files on different > machines and paths: > > ERROR: ovmf-edk2-stable202408-r0 do_prepare_recipe_sysroot: Error executing a python function in exec_func_python() autogenerated: > > The stack trace of python calls that resulted in this exception/failure was: > File: 'exec_func_python() autogenerated', lineno: 2, function: <module> > 0001: > *** 0002:extend_recipe_sysroot(d) > 0003: > File: '/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/classes-global/staging.bbclass', lineno: 624, function: extend_recipe_sysroot > 0620: > 0621: # Handle deferred binfiles > 0622: for l in binfiles: > 0623: (targetdir, dest) = binfiles[l] > *** 0624: staging_copyfile(l, targetdir, dest, postinsts, seendirs) > 0625: > 0626: bb.note("Installed into sysroot: %s" % str(msg_adding)) > 0627: bb.note("Skipping as already exists in sysroot: %s" % str(msg_exists)) > 0628: > File: '/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/classes-global/staging.bbclass', lineno: 165, function: staging_copyfile > 0161: os.symlink(linkto, dest) > 0162: #bb.warn(c) > 0163: else: > 0164: try: > *** 0165: os.link(c, dest) > 0166: except OSError as err: > 0167: if err.errno == errno.EXDEV: > 0168: bb.utils.copyfile(c, dest) > 0169: else: > Exception: FileExistsError: [Errno 17] File exists: '/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-667282/tmp/sysroots-components/x86_64/ovmf-native/usr/bin/edk2_basetools/BaseTools/Source/Python/AutoGen/__pycache__/WorkspaceAutoGen.cpython-312.pyc' -> '/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-667282/tmp/work/core2-64-poky-linux/ovmf/edk2-stable202408/recipe-sysroot-native/usr/bin/edk2_basetools/BaseTools/Source/Python/AutoGen/__pycache__/WorkspaceAutoGen.cpython-312.pyc' > > Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org> > --- > meta/recipes-core/ovmf/ovmf_git.bb | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/meta/recipes-core/ovmf/ovmf_git.bb b/meta/recipes-core/ovmf/ovmf_git.bb > index 202182397d..e761736453 100644 > --- a/meta/recipes-core/ovmf/ovmf_git.bb > +++ b/meta/recipes-core/ovmf/ovmf_git.bb > @@ -223,6 +223,7 @@ do_compile:class-target() { > > do_install:class-native() { > install -d ${D}/${bindir}/edk2_basetools > + find ${S}/BaseTools -name \*.pyc -exec rm -rf \{\} \; > cp -r ${S}/BaseTools ${D}/${bindir}/${EDK_TOOLS_DIR} > } > > -- > 2.34.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#205740): https://lists.openembedded.org/g/openembedded-core/message/205740 > Mute This Topic: https://lists.openembedded.org/mt/108998817/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Hi, On Mon, Oct 14, 2024 at 01:05:51PM +0200, Alexander Kanavin wrote: > This doesn't quite make sense. Why is the problem specific to .pyc > files and not other files ovmf installs? I don't know. At this point I'd rather remove all tests in uki support which are causing so many issues. I tried to dig deeper into ovmf native but it looks like a mess. meta-arm edk2 doesn't compile anything for native but just installs the python scripts and gets away with it. Same did not work for ovmf. Cheers, -Mikko
On Mon, 14 Oct 2024 at 13:21, Mikko Rapeli <mikko.rapeli@linaro.org> wrote: > I don't know. At this point I'd rather remove all tests in uki > support which are causing so many issues. > > I tried to dig deeper into ovmf native but it looks like a mess. > meta-arm edk2 doesn't compile anything for native but just installs > the python scripts and gets away with it. Same did not work for ovmf. The tests are important. Were you able to reproduce the issue locally? Ovmf is an awful component and a headache to work with, but I could take a look if I can get the same error. I'm just worried that deleting .pyc will only trigger the same error for the essential installed items. Alex
Hi, On Mon, Oct 14, 2024 at 01:28:07PM +0200, Alexander Kanavin wrote: > On Mon, 14 Oct 2024 at 13:21, Mikko Rapeli <mikko.rapeli@linaro.org> wrote: > > I don't know. At this point I'd rather remove all tests in uki > > support which are causing so many issues. > > > > I tried to dig deeper into ovmf native but it looks like a mess. > > meta-arm edk2 doesn't compile anything for native but just installs > > the python scripts and gets away with it. Same did not work for ovmf. > > The tests are important. > > Were you able to reproduce the issue locally? Ovmf is an awful > component and a headache to work with, but I could take a look if I > can get the same error. I'm just worried that deleting .pyc will only > trigger the same error for the essential installed items. No, I'm not able to reproduce the problem. Maybe I'd need to share the sstate to another machine or user to trigger this. Just changing build paths doesn't trigger the problem on my build machine. Alternatively I can remove all x86/ovmf using tests and make things work with u-boot firmware and aarch64, but then I need to test both qemuarm64 and genericarm64 since details differ between them. Cheers, -Mikko
On Mon, 14 Oct 2024 at 13:39, Mikko Rapeli <mikko.rapeli@linaro.org> wrote: > No, I'm not able to reproduce the problem. Maybe I'd need to share the sstate > to another machine or user to trigger this. Just changing build paths > doesn't trigger the problem on my build machine. Can I see the link to the autobuilder failure where this was observed? Alex
On Mon, 2024-10-14 at 13:05 +0200, Alexander Kanavin via lists.openembedded.org wrote: > This doesn't quite make sense. Why is the problem specific to .pyc > files and not other files ovmf installs? If it helps, I do roughly understand what is happening here and I believe this is the right fix. Cheers, Richard
On 14 Oct 2024, at 13:01, Richard Purdie via lists.openembedded.org <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote: > > On Mon, 2024-10-14 at 13:05 +0200, Alexander Kanavin via > lists.openembedded.org wrote: >> This doesn't quite make sense. Why is the problem specific to .pyc >> files and not other files ovmf installs? > > If it helps, I do roughly understand what is happening here and I > believe this is the right fix. So my understanding is that this is a general problem: if a native recipe ships a .py but not .pyc then when that script is used python will write a .pyc _into the sysroot_. If we then upgrade that recipe and it now ships a .pyc then you get this error, as the sysroot construction call wants to create a file that already exists. I think for scripts that are ran with python3-native then we should be shipping the .pyc files, even in native recipes, because then they’re pregenerated and will be used by python. The fun is when the script is ran by the host python, which is of indeterminate version. I wonder if python has a way to control where the files get written, so we could put freshly generated files into a separate directory that isn’t part of the sysroot? We could also make the sysroot read-only outside of the sysroot manipulation, which would force python to do something different too (though I believe this is just “don’t write pyc?). Note that this problem impacts plenty of other recipes that for historical reasons can't ship bytecode into the sysroot (see eg python3-setuptools), so I do think we need a general solution and not just a creeping and irreversible trend of deleting .pyc from the sysroot. Ross
On Mon, 2024-10-14 at 12:34 +0000, Ross Burton via lists.openembedded.org wrote: > On 14 Oct 2024, at 13:01, Richard Purdie via lists.openembedded.org > <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote: > > > > On Mon, 2024-10-14 at 13:05 +0200, Alexander Kanavin via > > lists.openembedded.org wrote: > > > This doesn't quite make sense. Why is the problem specific to > > > .pyc > > > files and not other files ovmf installs? > > > > If it helps, I do roughly understand what is happening here and I > > believe this is the right fix. > > So my understanding is that this is a general problem: if a native > recipe ships a .py but not .pyc then when that script is used python > will write a .pyc _into the sysroot_. If we then upgrade that recipe > and it now ships a .pyc then you get this error, as the sysroot > construction call wants to create a file that already exists. > > I think for scripts that are ran with python3-native then we should > be shipping the .pyc files, even in native recipes, because then > they’re pregenerated and will be used by python. The fun is when the > script is ran by the host python, which is of indeterminate version. For fun, there are things which could be called by either the host python or python3-native from the sysroot depending on whether that was added to PATH for other reasons. That complicates things somewhat. > I wonder if python has a way to control where the files get written, > so we could put freshly generated files into a separate directory > that isn’t part of the sysroot? We could also make the sysroot read- > only outside of the sysroot manipulation, which would force python to > do something different too (though I believe this is just “don’t > write pyc?). > > Note that this problem impacts plenty of other recipes that for > historical reasons can't ship bytecode into the sysroot (see eg > python3-setuptools), so I do think we need a general solution and not > just a creeping and irreversible trend of deleting .pyc from the > sysroot. I'm torn. It is a complex problem causing issues for patches which are otherwise ok and requires a great deal of thought to fix differently. The performance impact of python generating those files at first run probably is negligible, compared even to the overhead of extracting files from sstate. Is this an issue we want to focus effort on? Are there gains here to be had? Cheers, Richard
diff --git a/meta/recipes-core/ovmf/ovmf_git.bb b/meta/recipes-core/ovmf/ovmf_git.bb index 202182397d..e761736453 100644 --- a/meta/recipes-core/ovmf/ovmf_git.bb +++ b/meta/recipes-core/ovmf/ovmf_git.bb @@ -223,6 +223,7 @@ do_compile:class-target() { do_install:class-native() { install -d ${D}/${bindir}/edk2_basetools + find ${S}/BaseTools -name \*.pyc -exec rm -rf \{\} \; cp -r ${S}/BaseTools ${D}/${bindir}/${EDK_TOOLS_DIR} }
They break builds which share sstate files on different machines and paths: ERROR: ovmf-edk2-stable202408-r0 do_prepare_recipe_sysroot: Error executing a python function in exec_func_python() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001: *** 0002:extend_recipe_sysroot(d) 0003: File: '/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/classes-global/staging.bbclass', lineno: 624, function: extend_recipe_sysroot 0620: 0621: # Handle deferred binfiles 0622: for l in binfiles: 0623: (targetdir, dest) = binfiles[l] *** 0624: staging_copyfile(l, targetdir, dest, postinsts, seendirs) 0625: 0626: bb.note("Installed into sysroot: %s" % str(msg_adding)) 0627: bb.note("Skipping as already exists in sysroot: %s" % str(msg_exists)) 0628: File: '/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/classes-global/staging.bbclass', lineno: 165, function: staging_copyfile 0161: os.symlink(linkto, dest) 0162: #bb.warn(c) 0163: else: 0164: try: *** 0165: os.link(c, dest) 0166: except OSError as err: 0167: if err.errno == errno.EXDEV: 0168: bb.utils.copyfile(c, dest) 0169: else: Exception: FileExistsError: [Errno 17] File exists: '/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-667282/tmp/sysroots-components/x86_64/ovmf-native/usr/bin/edk2_basetools/BaseTools/Source/Python/AutoGen/__pycache__/WorkspaceAutoGen.cpython-312.pyc' -> '/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-667282/tmp/work/core2-64-poky-linux/ovmf/edk2-stable202408/recipe-sysroot-native/usr/bin/edk2_basetools/BaseTools/Source/Python/AutoGen/__pycache__/WorkspaceAutoGen.cpython-312.pyc' Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org> --- meta/recipes-core/ovmf/ovmf_git.bb | 1 + 1 file changed, 1 insertion(+)