| Message ID | 20231023110618.3525311-3-alex@linutronix.de |
|---|---|
| State | New |
| Headers | show |
| Series | [1/3] selftest/sstatetests: add tests for 'bitbake -S printdiff' | expand |
Hello Alex, On Mon, 23 Oct 2023 13:06:18 +0200 "Alexander Kanavin" <alex.kanavin@gmail.com> wrote: > Specifically, the test checks that everything needed for building > standard oe-core images for x86_64 and arm64 is available from > the cache (with minor exceptions). Going forward, a complete > world check could be enabled and additional configurations, > but that requires improvements to performance of hash equivalence > server in particular. > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> This is triggering: AssertionError: False is not true : Missing objects in the cache: weston_12.0.2.bb:do_package_write_ipk weston_12.0.2.bb:do_package_write_deb https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6091/steps/23/logs/stdio Luca
On Tue, 24 Oct 2023 at 10:33, Luca Ceresoli <luca.ceresoli@bootlin.com> wrote: > This is triggering: > > AssertionError: False is not true : Missing objects in the cache: > weston_12.0.2.bb:do_package_write_ipk > weston_12.0.2.bb:do_package_write_deb > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6091/steps/23/logs/stdio This is for qemux86_64 and seems to indicate that world builds (with multilib enabled) don't match plain builds (which don't build core-image-weston). I'll resubmit with that removed. Alex
On Thu, 2023-10-26 at 12:26 +0200, Alexander Kanavin wrote: > On Tue, 24 Oct 2023 at 10:33, Luca Ceresoli <luca.ceresoli@bootlin.com> wrote: > > This is triggering: > > > > AssertionError: False is not true : Missing objects in the cache: > > weston_12.0.2.bb:do_package_write_ipk > > weston_12.0.2.bb:do_package_write_deb > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6091/steps/23/logs/stdio > > This is for qemux86_64 and seems to indicate that world builds (with > multilib enabled) don't match plain builds (which don't build > core-image-weston). I'll resubmit with that removed. Can we file a bug to look into that? I'm also a little puzzled as this does seem to work some of the time as I managed green builds of this. Thanks, Richard
On Thu, 26 Oct 2023 at 12:53, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > Can we file a bug to look into that? > > I'm also a little puzzled as this does seem to work some of the time as > I managed green builds of this. Filed: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15253 We need to think of ways to test/debug this that don't require running a-full. Alex
On Thu, 2023-10-26 at 13:16 +0200, Alexander Kanavin wrote: > On Thu, 26 Oct 2023 at 12:53, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > Can we file a bug to look into that? > > > > I'm also a little puzzled as this does seem to work some of the time as > > I managed green builds of this. > > Filed: > https://bugzilla.yoctoproject.org/show_bug.cgi?id=15253 > > We need to think of ways to test/debug this that don't require running a-full. We can create a separate target (builder) on the AB to just run the test? Cheers, Richard
On Thu, 26 Oct 2023 at 13:26, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > We need to think of ways to test/debug this that don't require running a-full. > > We can create a separate target (builder) on the AB to just run the > test? But it needs to run just after the task (e.g. world or some other which should be dynamically specified) that is supposed to create the required objects. I guess it needs to be done with a custom branch on yocto-autobuilder-helper which would define that, but then how to trigger it through the UI? Alex
On Thu, 2023-10-26 at 13:32 +0200, Alexander Kanavin wrote: > On Thu, 26 Oct 2023 at 13:26, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > We need to think of ways to test/debug this that don't require running a-full. > > > > We can create a separate target (builder) on the AB to just run the > > test? > > But it needs to run just after the task (e.g. world or some other > which should be dynamically specified) that is supposed to create the > required objects. I guess it needs to be done with a custom branch on > yocto-autobuilder-helper which would define that, but then how to > trigger it through the UI? We could run the appropriate build and then the test in the next step? It could be a standalone builder target which just isn't triggered by default? Not all targets have to be triggered by something day to day. Cheers, Richard
On Thu, 2023-10-26 at 13:32 +0200, Alexander Kanavin wrote: > On Thu, 26 Oct 2023 at 13:26, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > We need to think of ways to test/debug this that don't require running a-full. > > > > We can create a separate target (builder) on the AB to just run the > > test? > > But it needs to run just after the task (e.g. world or some other > which should be dynamically specified) that is supposed to create the > required objects. I guess it needs to be done with a custom branch on > yocto-autobuilder-helper which would define that, but then how to > trigger it through the UI? https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6105/steps/25/logs/stdio AssertionError: False is not true : Missing objects in the cache: connman-gnome_0.7.bb:do_package_write_rpm font-util_1.4.1.bb:do_package_write_rpm gdk-pixbuf_2.42.10.bb:do_package_write_rpm librsvg_2.56.3.bb:do_package_write_rpm gstreamer1.0-plugins-bad_1.22.6.bb:do_package_write_rpm gst-examples_1.18.6.bb:do_package_write_rpm which is a little worrying :/ Cheers, Richard
On Thu, 26 Oct 2023 at 14:53, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6105/steps/25/logs/stdio > > AssertionError: False is not true : Missing objects in the cache: > connman-gnome_0.7.bb:do_package_write_rpm > font-util_1.4.1.bb:do_package_write_rpm > gdk-pixbuf_2.42.10.bb:do_package_write_rpm > librsvg_2.56.3.bb:do_package_write_rpm > gstreamer1.0-plugins-bad_1.22.6.bb:do_package_write_rpm > gst-examples_1.18.6.bb:do_package_write_rpm > > which is a little worrying :/ I've seen this in local testing some days back too, with the same set of missing objects, but I had ascribed it to master not being populated yet. Seems like it's not that simple, but I don't have a clear plan forward right now. Alex
On Thu, 2023-10-26 at 16:36 +0200, Alexander Kanavin wrote: > On Thu, 26 Oct 2023 at 14:53, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6105/steps/25/logs/stdio > > > > AssertionError: False is not true : Missing objects in the cache: > > connman-gnome_0.7.bb:do_package_write_rpm > > font-util_1.4.1.bb:do_package_write_rpm > > gdk-pixbuf_2.42.10.bb:do_package_write_rpm > > librsvg_2.56.3.bb:do_package_write_rpm > > gstreamer1.0-plugins-bad_1.22.6.bb:do_package_write_rpm > > gst-examples_1.18.6.bb:do_package_write_rpm > > > > which is a little worrying :/ > > I've seen this in local testing some days back too, with the same set > of missing objects, but I had ascribed it to master not being > populated yet. Seems like it's not that simple, but I don't have a > clear plan forward right now. Lets make the test print the hashes it can't find? It would be useful to know if they're perhaps delayed somehow on the CDN? Seems odd it is the same list of things though. We need to find the hashes we build, and the ones we can't find and see if they're the same. If we get the siginfo files for them, we can compare and see why they're changing. Cheers, Richard
On Thu, 26 Oct 2023 at 18:13, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > Lets make the test print the hashes it can't find? > > It would be useful to know if they're perhaps delayed somehow on the > CDN? > > Seems odd it is the same list of things though. > > We need to find the hashes we build, and the ones we can't find and see > if they're the same. If we get the siginfo files for them, we can > compare and see why they're changing. I've just submitted a modified patch for the test that prints hashes that were requested if the test fails. I've also done two run-throughs of the standalone builder for it: https://autobuilder.yoctoproject.org/typhoon/#/builders/158 The first one failed (which was expected), then I ran two standalone qemuarm64/qemux86-64 jobs (which did actually rebuild the missing stuff like perf etc.), and the second run succeeded. So let's see it fail mysteriously again in a-full or a-quick; there should now be more info to try and dig into that. Alex
diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py index b96eacc9ad8..e895f66360c 100644 --- a/meta/lib/oeqa/selftest/cases/sstatetests.py +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py @@ -14,6 +14,7 @@ import re from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer, get_bb_vars from oeqa.selftest.case import OESelftestTestCase +from oeqa.core.decorator import OETestTag import oe import bb.siggen @@ -886,3 +887,49 @@ expected_sametmp_output, expected_difftmp_output) INHERIT += "base-do-configure-modified" """, expected_sametmp_output, expected_difftmp_output) + +@OETestTag("yocto-mirrors") +class SStateMirrors(SStateBase): + def check_bb_output(self, output, exceptions): + in_tasks = False + missing_objects = [] + for l in output.splitlines(): + if "The differences between the current build and any cached tasks start at the following tasks" in l: + in_tasks = True + continue + if "Writing task signature files" in l: + in_tasks = False + continue + if in_tasks: + recipe_task = l.split("/")[-1] + recipe, task = recipe_task.split(":") + for e in exceptions: + if e[0] in recipe and task == e[1]: + break + else: + missing_objects.append(recipe_task) + self.assertTrue(len(missing_objects) == 0, "Missing objects in the cache:\n{}".format("\n".join(missing_objects))) + + + def run_test_cdn_mirror(self, machine, targets, exceptions): + exceptions = exceptions + [[t, "do_deploy_source_date_epoch"] for t in targets.split()] + exceptions = exceptions + [[t, "do_image_qa"] for t in targets.split()] + self.config_sstate(True) + self.append_config(""" +MACHINE = "{}" +BB_HASHSERVE_UPSTREAM = "hashserv.yocto.io:8687" +SSTATE_MIRRORS ?= "file://.* http://cdn.jsdelivr.net/yocto/sstate/all/PATH;downloadfilename=PATH" +""".format(machine)) + result = bitbake("-S printdiff {}".format(targets)) + self.check_bb_output(result.output, exceptions) + + def test_cdn_mirror_qemux86_64(self): + # Example: + # exceptions = [ ["packagegroup-core-sdk","do_package"] ] + exceptions = [] + self.run_test_cdn_mirror("qemux86-64", "core-image-minimal core-image-full-cmdline core-image-weston core-image-sato-sdk", exceptions) + + def test_cdn_mirror_qemuarm64(self): + exceptions = [] + # core-image-weston isn't produced for arm64 currently + self.run_test_cdn_mirror("qemuarm64", "core-image-minimal core-image-full-cmdline core-image-sato-sdk", exceptions)
Specifically, the test checks that everything needed for building standard oe-core images for x86_64 and arm64 is available from the cache (with minor exceptions). Going forward, a complete world check could be enabled and additional configurations, but that requires improvements to performance of hash equivalence server in particular. Signed-off-by: Alexander Kanavin <alex@linutronix.de> --- meta/lib/oeqa/selftest/cases/sstatetests.py | 47 +++++++++++++++++++++ 1 file changed, 47 insertions(+)