| Message ID | 20241028034618.1504830-1-vince_chang@aspeedtech.com |
|---|---|
| Headers | show |
| Series | wic support different sector size | expand |
These two patches have merged, and unfortunately there are now failures on the autobuilder: https://valkyrie.yoctoproject.org/#/builders/35/builds/346/steps/14/logs/stdio https://valkyrie.yoctoproject.org/#/builders/54/builds/325/steps/14/logs/stdio Alex On Mon, 28 Oct 2024 at 04:46, Vince Chang via lists.openembedded.org <vince_chang=aspeedtech.com@lists.openembedded.org> wrote: > > v1: > - wic support different sector size > - util-linux: sfdisk sector size improvements (merged) > > v2: > - For "wic ls" modify get_partitions() to support WIC_SECTOR_SIZE. > - Add oe-selftest for WIC_SECTOR_SIZE. > > Vince Chang (2): > wic: add WIC_SECTOR_SIZE variable > selftest: wic: add test for WIC_SECTOR_SIZE > > meta/lib/oeqa/selftest/cases/wic.py | 50 +++++++++++++++++++++ > scripts/lib/wic/engine.py | 18 +++++++- > scripts/lib/wic/plugins/imager/direct.py | 56 ++++++++++++++---------- > 3 files changed, 100 insertions(+), 24 deletions(-) > > -- > 2.25.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#206417): https://lists.openembedded.org/g/openembedded-core/message/206417 > Mute This Topic: https://lists.openembedded.org/mt/109251303/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Hi Alex, On Wed, 30 Oct 2024 at 21:01, Alexander Kanavin via lists.openembedded.org > These two patches have merged, and unfortunately there are now > failures on the autobuilder: > https://valkyrie.yoctoproject.org/#/builders/35/builds/346/steps/14/logs/stdio > https://valkyrie.yoctoproject.org/#/builders/54/builds/325/steps/14/logs/stdio It's strange that this line of code fails to execute. It works fine on my build machine. Is there a way to find out which version of parted is being used on the autobuilder? File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/wic.py", line 877, in test_wic_sector_size res = runCmd("export PARTED_SECTOR_SIZE=%d; parted -m %s unit b p 2>/dev/null" % (wic_sector_size, images[0])) File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 212, in runCmd raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output)) AssertionError: Command 'export PARTED_SECTOR_SIZE=4096; parted -m /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3537958/wic-tmp/tmpr6vrw11i-202410301331-sda.direct unit b p 2>/dev/null' returned non-zero exit status 127: Thanks, Vince
On Wed, Oct 30, 2024 at 7:11 PM Vince Chang via lists.openembedded.org <vince_chang=aspeedtech.com@lists.openembedded.org> wrote: > > Hi Alex, > > On Wed, 30 Oct 2024 at 21:01, Alexander Kanavin via lists.openembedded.org > > These two patches have merged, and unfortunately there are now > > failures on the autobuilder: > > https://valkyrie.yoctoproject.org/#/builders/35/builds/346/steps/14/logs/stdio > > https://valkyrie.yoctoproject.org/#/builders/54/builds/325/steps/14/logs/stdio > > It's strange that this line of code fails to execute. It works fine on my build machine. Is there a way to find out which version of parted is being used on the autobuilder? > > File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/wic.py", line 877, in test_wic_sector_size > res = runCmd("export PARTED_SECTOR_SIZE=%d; parted -m %s unit b p 2>/dev/null" % (wic_sector_size, images[0])) > File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 212, in runCmd > raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output)) > AssertionError: Command 'export PARTED_SECTOR_SIZE=4096; parted -m /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3537958/wic-tmp/tmpr6vrw11i-202410301331-sda.direct unit b p 2>/dev/null' returned non-zero exit status 127: > perhaps make it PARTED_SECTOR_SIZE=%d parted -s %s mklabel %s" % (self.sector_size, device, ptable_format), self.native_sysroot) > Thanks, > Vince > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#206566): https://lists.openembedded.org/g/openembedded-core/message/206566 > Mute This Topic: https://lists.openembedded.org/mt/109308341/1997914 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Thu, 31 Oct 2024 at 03:46, Khem Raj <raj.khem@gmail.com> wrote: > > File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/wic.py", line 877, in test_wic_sector_size > > res = runCmd("export PARTED_SECTOR_SIZE=%d; parted -m %s unit b p 2>/dev/null" % (wic_sector_size, images[0])) > > File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 212, in runCmd > > raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output)) > > AssertionError: Command 'export PARTED_SECTOR_SIZE=4096; parted -m /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3537958/wic-tmp/tmpr6vrw11i-202410301331-sda.direct unit b p 2>/dev/null' returned non-zero exit status 127: > > > > perhaps make it > PARTED_SECTOR_SIZE=%d parted -s %s mklabel %s" % (self.sector_size, > device, ptable_format), self.native_sysroot) I tried this (dropping 2>/dev/null), and indeed the error is no longer obscured: AssertionError: Command 'export PARTED_SECTOR_SIZE=4096; parted -m /home/akanavin/poky/build-st/wic-tmp/tmprac0npku-202410311007-sda.direct unit b p' returned non-zero exit status 127: /bin/sh: 1: parted: not found So basically the test forgets to adjust the PATH to include wic-tools, like others wic tests (that need to run parted) do. Some build hosts have it, others don't, so the error is intermittent. Also if you can (in a separate commit) drop all usage of this "2>/dev/null" pattern in wic tests that only harms fail diagnostics, that would be helpful. Alex
Hi Alex, On Thu, 31 Oct 2024 at 5:18 PM, Alexander Kanavin wrote: > Also if you can (in a separate commit) drop all usage of this > "2>/dev/null" pattern in wic tests that only harms fail diagnostics, > that would be helpful. I will send a separate commit to remove "2>/dev/null" in the wic test cases. Thanks, Vince
Hi Alex, > On Thu, 31 Oct 2024 at 5:18 PM, Alexander Kanavin wrote: > > > Also if you can (in a separate commit) drop all usage of this > > "2>/dev/null" pattern in wic tests that only harms fail diagnostics, > > that would be helpful. > > I will send a separate commit to remove "2>/dev/null" in the wic test cases. > The wic test cases use out.splitlines to get the command result. If we remove "2>/dev/null", it may capture other warning messages, making it unable to get the current result. To avoid needing additional code to use splitted.index("BYT;") and considering that ‘2>/dev/null’ is also used in ‘debugfs’, I would suggest not removing "2>/dev/null". If the same issue arises again, we can manually remove it to see where the error is. What do you think? Thanks, Vince
On Fri, 1 Nov 2024 at 03:57, Vince Chang <vince_chang@aspeedtech.com> wrote: > > > Also if you can (in a separate commit) drop all usage of this > > > "2>/dev/null" pattern in wic tests that only harms fail diagnostics, > > > that would be helpful. > > > > I will send a separate commit to remove "2>/dev/null" in the wic test cases. > > > > The wic test cases use out.splitlines to get the command result. If we remove "2>/dev/null", it may capture other warning messages, making it unable to get the current result. > To avoid needing additional code to use splitted.index("BYT;") and considering that ‘2>/dev/null’ is also used in ‘debugfs’, I would suggest not removing "2>/dev/null". If the same issue arises again, we can manually remove it to see where the error is. What do you think? > If the issue is intermittent or difficult to reproduce (and in this case it was - I had to spend a couple hours reproducing it for you on a particular build host in yocto CI cluster), it is far better to write the code such that the needed diagnostics are printed right when the error occurs in the first place in CI. Anyway, implementation runCmd() indeed defaults to stderr being redirected and appended to stdout, so the standard stream and the error stream are mashed together. To avoid that, you should also set stderr=subprocess.PIPE when calling runCmd(). Check meta/lib/oeqa/selftest/cases/runcmd.py for examples: def test_output(self): result = runCmd("echo stdout; echo stderr >&2", shell=True, sync=False) self.assertEqual("stdout\nstderr", result.output) self.assertEqual("", result.error) def test_output_split(self): result = runCmd("echo stdout; echo stderr >&2", shell=True, stderr=subprocess.PIPE, sync=False) self.assertEqual("stdout", result.output) self.assertEqual("stderr", result.error) Alex
Hi Alex, On Fri, 1 Nov 2024 at 12:32 PM, Alexander Kanavin wrote: > Anyway, implementation runCmd() indeed defaults to stderr being > redirected and appended to stdout, so the standard stream and the > error stream are mashed together. > > To avoid that, you should also set stderr=subprocess.PIPE when calling > runCmd(). Check meta/lib/oeqa/selftest/cases/runcmd.py for examples: > > def test_output(self): > result = runCmd("echo stdout; echo stderr >&2", shell=True, > sync=False) > self.assertEqual("stdout\nstderr", result.output) > self.assertEqual("", result.error) > > def test_output_split(self): > result = runCmd("echo stdout; echo stderr >&2", shell=True, > stderr=subprocess.PIPE, sync=False) > self.assertEqual("stdout", result.output) > self.assertEqual("stderr", result.error) I didn't realize that stderr=subprocess.PIPE could be used. I've submitted a separate commit. https://lists.openembedded.org/g/openembedded-core/message/206602 Thanks, Vince
Thanks! But you should still fix the original fail I think? Alex On Fri 1. Nov 2024 at 8.37, Vince Chang <vince_chang@aspeedtech.com> wrote: > Hi Alex, > > On Fri, 1 Nov 2024 at 12:32 PM, Alexander Kanavin wrote: > > Anyway, implementation runCmd() indeed defaults to stderr being > > redirected and appended to stdout, so the standard stream and the > > error stream are mashed together. > > > > To avoid that, you should also set stderr=subprocess.PIPE when calling > > runCmd(). Check meta/lib/oeqa/selftest/cases/runcmd.py for examples: > > > > def test_output(self): > > result = runCmd("echo stdout; echo stderr >&2", shell=True, > > sync=False) > > self.assertEqual("stdout\nstderr", result.output) > > self.assertEqual("", result.error) > > > > def test_output_split(self): > > result = runCmd("echo stdout; echo stderr >&2", shell=True, > > stderr=subprocess.PIPE, sync=False) > > self.assertEqual("stdout", result.output) > > self.assertEqual("stderr", result.error) > > I didn't realize that stderr=subprocess.PIPE could be used. I've submitted > a separate commit. > https://lists.openembedded.org/g/openembedded-core/message/206602 > > Thanks, > Vince >
Hi Alex, Yes, I have submitted a commit to add wic-tools into the PATH. https://lists.openembedded.org/g/openembedded-core/message/206607 Thanks, Vince From: Alexander Kanavin <alex.kanavin@gmail.com> Sent: Friday, November 1, 2024 3:39 PM To: Vince Chang <vince_chang@aspeedtech.com> Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; openembedded-core@lists.openembedded.org; Jamin Lin <jamin_lin@aspeedtech.com>; Troy Lee <troy_lee@aspeedtech.com>; Khem Raj <raj.khem@gmail.com> Subject: Re: [OE-core] [PATCH v2 0/2] wic support different sector size Thanks! But you should still fix the original fail I think? Alex On Fri 1. Nov 2024 at 8.37, Vince Chang <vince_chang@aspeedtech.com<mailto:vince_chang@aspeedtech.com>> wrote: Hi Alex, On Fri, 1 Nov 2024 at 12:32 PM, Alexander Kanavin wrote: > Anyway, implementation runCmd() indeed defaults to stderr being > redirected and appended to stdout, so the standard stream and the > error stream are mashed together. > > To avoid that, you should also set stderr=subprocess.PIPE when calling > runCmd(). Check meta/lib/oeqa/selftest/cases/runcmd.py for examples: > > def test_output(self): > result = runCmd("echo stdout; echo stderr >&2", shell=True, > sync=False) > self.assertEqual("stdout\nstderr", result.output) > self.assertEqual("", result.error) > > def test_output_split(self): > result = runCmd("echo stdout; echo stderr >&2", shell=True, > stderr=subprocess.PIPE, sync=False) > self.assertEqual("stdout", result.output) > self.assertEqual("stderr", result.error) I didn't realize that stderr=subprocess.PIPE could be used. I've submitted a separate commit. https://lists.openembedded.org/g/openembedded-core/message/206602 Thanks, Vince
On Fri, 1 Nov 2024 at 08:37, Vince Chang <vince_chang@aspeedtech.com> wrote: > I didn't realize that stderr=subprocess.PIPE could be used. I've submitted a separate commit. > https://lists.openembedded.org/g/openembedded-core/message/206602 I just added one more fix that makes runCmd actually print stderr when it's split from stdout and an error occurs: https://lists.openembedded.org/g/openembedded-core/message/206610 and with that and your two fixes everything should be working correctly. Alex