From patchwork Fri Jun 9 06:47:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alexis_Lothor=C3=A9?= X-Patchwork-Id: 25317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34641C7EE37 for ; Fri, 9 Jun 2023 06:47:57 +0000 (UTC) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx.groups.io with SMTP id smtpd.web11.7419.1686293270649314470 for ; Thu, 08 Jun 2023 23:47:51 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=GNNV5d3m; spf=pass (domain: bootlin.com, ip: 217.70.183.195, mailfrom: alexis.lothore@bootlin.com) X-GND-Sasl: alexis.lothore@bootlin.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1686293268; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KBCw1B/O8Hp0O6KVv5y7sU9oxjcHgbOlO/DG8skt1Vs=; b=GNNV5d3mHmxtK4RyWDRLMYRE5hPagpXDA5UH1TeIFP4+Lw/2zkKPT8+yO/NtwEFVAw8YhL hU1KT7++qs5X43oGcSS/nAwuB93xn4sXDIQIoOtsxnhrs5bC8ve2EmLSEtVEI0Fdso+j8O 8+eo0Q8H10+8BptUT7W2lE0XU/whbMhtadvi6qq0G9FwqllrSgSNUHHyOIBakoUIjy9LtJ WCzP+xxil8GWnsqwSi6wTuxKABjx+deYsrXbY8kz/MINr6sI+F+gVCfV1df+n7zmJXVX3t yNOnUQBgXxXRx76sDnyT0t0cvcNJtGB4XLW83wgO+KYMh5+f1qCVBP19zHYLgw== X-GND-Sasl: alexis.lothore@bootlin.com X-GND-Sasl: alexis.lothore@bootlin.com X-GND-Sasl: alexis.lothore@bootlin.com Received: by mail.gandi.net (Postfix) with ESMTPSA id 2F13860008; Fri, 9 Jun 2023 06:47:48 +0000 (UTC) From: alexis.lothore@bootlin.com To: Cc: Thomas Petazzoni , Alexandre Belloni , =?utf-8?q?Alexis_Lothor?= =?utf-8?q?=C3=A9?= Subject: [OE-Core][PATCH v3 1/4] oeqa/core/runner: add helper to know about expected failures Date: Fri, 9 Jun 2023 08:47:59 +0200 Message-Id: <20230609064802.11777-2-alexis.lothore@bootlin.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230609064802.11777-1-alexis.lothore@bootlin.com> References: <20230609064802.11777-1-alexis.lothore@bootlin.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 09 Jun 2023 06:47:57 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/182537 From: Alexis Lothoré Testing framework currently uses the unittest.expectedFailure decorator for tests that can have intermittent failures (see PTEST_EXPECT_FAILURE = "1") in core-image-ptest.bb. While it allows upper layers to run tests without failing on "fragile" tests, it prevents those from knowing more about those failing tests since they are not accounting as failures (for example we could want to retrieve some logs about failed tests to improve them, and eventually to drop expectFailure decorator) Add a helper to allow upper layers to know about those failures which won't make global testing session Signed-off-by: Alexis Lothoré --- meta/lib/oeqa/core/runner.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/meta/lib/oeqa/core/runner.py b/meta/lib/oeqa/core/runner.py index d50690ab37f8..5077eb8e3e32 100644 --- a/meta/lib/oeqa/core/runner.py +++ b/meta/lib/oeqa/core/runner.py @@ -229,6 +229,10 @@ class OETestResult(_TestResult): # Override as we unexpected successes aren't failures for us return (len(self.failures) == len(self.errors) == 0) + def hasAnyFailingTest(self): + # Account for expected failures + return not self.wasSuccessful() or len(self.expectedFailures) + class OEListTestsResult(object): def wasSuccessful(self): return True From patchwork Fri Jun 9 06:48:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alexis_Lothor=C3=A9?= X-Patchwork-Id: 25316 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 220BDC7EE29 for ; Fri, 9 Jun 2023 06:47:57 +0000 (UTC) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx.groups.io with SMTP id smtpd.web10.7350.1686293270819606344 for ; Thu, 08 Jun 2023 23:47:51 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=nhKtDoqa; spf=pass (domain: bootlin.com, ip: 217.70.183.195, mailfrom: alexis.lothore@bootlin.com) X-GND-Sasl: alexis.lothore@bootlin.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1686293268; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7IAJNYgkrVsfDpP7lCSbRWA8OmWIg7KNDS7TSafLzfI=; b=nhKtDoqavzMAT5Dcbmp84M3XGc5xq5HdGBirtQxxz5dBT4lN6nxWIVP+yMG4PTSYpBcitA uDbPl+vUDWYHaC1+CF6NNOoYt+UgJ+HZhsMH2U7rI9jnuuYSb/ErtENVmflolYVx38Nd/X qUkX/J7nmKGMe7IpPXAGZK9knHtSG48KH4LgEHuyggiWrA8gBSeeH7kWC1M8U4v9pL6xiR vqIV0SmkMwnpTF0CHVxRrXVnPblMiBDr6thcOWwnbLhkOTu0se52gJUDR7jKEJF39/O4Yt ksZ4n2Xu1Gc59JuA6atSyIFw9ZpJdshgK6pLKWifjrji2+5LwFIhaMMe1RVeFg== X-GND-Sasl: alexis.lothore@bootlin.com X-GND-Sasl: alexis.lothore@bootlin.com X-GND-Sasl: alexis.lothore@bootlin.com Received: by mail.gandi.net (Postfix) with ESMTPSA id A803160002; Fri, 9 Jun 2023 06:47:48 +0000 (UTC) From: alexis.lothore@bootlin.com To: Cc: Thomas Petazzoni , Alexandre Belloni , =?utf-8?q?Alexis_Lothor?= =?utf-8?q?=C3=A9?= Subject: [OE-Core][PATCH v3 2/4] oeqa/target/ssh: update options for SCP Date: Fri, 9 Jun 2023 08:48:00 +0200 Message-Id: <20230609064802.11777-3-alexis.lothore@bootlin.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230609064802.11777-1-alexis.lothore@bootlin.com> References: <20230609064802.11777-1-alexis.lothore@bootlin.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 09 Jun 2023 06:47:57 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/182534 From: Alexis Lothoré By default scp expects files. Passing -r option allows to copy directories too Signed-off-by: Alexis Lothoré --- Changes since v1: - drop legacy scp protocol option --- meta/lib/oeqa/core/target/ssh.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/meta/lib/oeqa/core/target/ssh.py b/meta/lib/oeqa/core/target/ssh.py index 51079075b5bd..e650302052db 100644 --- a/meta/lib/oeqa/core/target/ssh.py +++ b/meta/lib/oeqa/core/target/ssh.py @@ -40,8 +40,11 @@ class OESSHTarget(OETarget): '-o', 'StrictHostKeyChecking=no', '-o', 'LogLevel=ERROR' ] + scp_options = [ + '-r' + ] self.ssh = ['ssh', '-l', self.user ] + ssh_options - self.scp = ['scp'] + ssh_options + self.scp = ['scp'] + ssh_options + scp_options if port: self.ssh = self.ssh + [ '-p', port ] self.scp = self.scp + [ '-P', port ] From patchwork Fri Jun 9 06:48:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alexis_Lothor=C3=A9?= X-Patchwork-Id: 25318 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 347AFC83005 for ; Fri, 9 Jun 2023 06:47:57 +0000 (UTC) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx.groups.io with SMTP id smtpd.web10.7349.1686293270744409048 for ; Thu, 08 Jun 2023 23:47:51 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=kFuJNJcI; spf=pass (domain: bootlin.com, ip: 217.70.183.195, mailfrom: alexis.lothore@bootlin.com) X-GND-Sasl: alexis.lothore@bootlin.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1686293269; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xPe1iucLq1Gp7IPsUSMyOZRk+yc+lnR44Tt42jR6KoM=; b=kFuJNJcIsiZ43AKFkEjhEDbB9zc12v2R528in4rOd4LMla0w5gyw0QPh67mFpJX62/5RFI k2rioqykssg0Y60TGrxNxCohf4Jr5QYIFT6qDxxHV5+ktVwJTaOEyOnLcyXkRcY+LN5KX6 qOhvr4HXy1PIiSVAXsco5bFbMTOLmNQfNMFYV7rQ3fo1bLWGQTi9GDnJX7QXR9+iUsbPxU f54c+S6xACnmXGYWaQeMAJYXITup6zTyubb//V0Bofhu7lWXYVwyVh+28WpJZENio3+QVw 4SiQGN4ECdJN7O651pBULyV+cETcXYwZGsm+kntWYLjJxZYytyRSphLHRBTlZw== X-GND-Sasl: alexis.lothore@bootlin.com X-GND-Sasl: alexis.lothore@bootlin.com X-GND-Sasl: alexis.lothore@bootlin.com Received: by mail.gandi.net (Postfix) with ESMTPSA id 1A1866000A; Fri, 9 Jun 2023 06:47:49 +0000 (UTC) From: alexis.lothore@bootlin.com To: Cc: Thomas Petazzoni , Alexandre Belloni , =?utf-8?q?Alexis_Lothor?= =?utf-8?q?=C3=A9?= Subject: [OE-Core][PATCH v3 3/4] testimage: implement test artifacts retriever for failing tests Date: Fri, 9 Jun 2023 08:48:01 +0200 Message-Id: <20230609064802.11777-4-alexis.lothore@bootlin.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230609064802.11777-1-alexis.lothore@bootlin.com> References: <20230609064802.11777-1-alexis.lothore@bootlin.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 09 Jun 2023 06:47:57 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/182535 From: Alexis Lothoré Add a basic artifacts retrievers in testimage class which: - triggers when at least one runtime test fails but tests execution encountered no major issue - reads a list of paths to retrieve from TESTIMAGE_FAILED_QA_ARTIFACTS - checks for artifacts presence on target - retrieve those files over scp thanks to existing ssh class - store those files in an "artifacts" directory in "tmp/log/oeqa/" This implementation assumes that the SSH or Qemu target has run and finished gracefully. If tests do not finish because of an exception, artifacts will not be retrieved Bring partial solution to [YOCTO #14901] Signed-off-by: Alexis Lothoré --- Changes since v2: - remove ptests directories from default artifact list, now managed in core-image-ptest.bb - use new function to check for any failed ptest Changes since v1: - only gather artifacts in nominal case (ie qemu runs without any raised exception) - list artifacts directly in variable instead of using external file - use standard variables in artifacts paths - allow glob patterns usage in artifacts paths - expand/filter artifacts list on target before retrieving them - tune default artifacts list --- meta/classes-recipe/testimage.bbclass | 48 +++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/meta/classes-recipe/testimage.bbclass b/meta/classes-recipe/testimage.bbclass index b48cd96575d2..765184c18008 100644 --- a/meta/classes-recipe/testimage.bbclass +++ b/meta/classes-recipe/testimage.bbclass @@ -18,6 +18,15 @@ inherit image-artifact-names TESTIMAGE_AUTO ??= "0" +# When any test fails, TESTIMAGE_FAILED_QA ARTIFACTS will be parsed and for +# each entry in it, if artifact pointed by path description exists on target, +# it will be retrieved onto host + +TESTIMAGE_FAILED_QA_ARTIFACTS ??= "\ + ${localstatedir}/log \ + ${sysconfdir}/version \ + ${sysconfdir}/os-release" + # You can set (or append to) TEST_SUITES in local.conf to select the tests # which you want to run for your target. # The test names are the module names in meta/lib/oeqa/runtime/cases. @@ -192,6 +201,39 @@ def get_testimage_boot_patterns(d): boot_patterns[flag] = flagval.encode().decode('unicode-escape') return boot_patterns +def get_artifacts_list(target, raw_list): + result = [] + # Passed list may contains patterns in paths, expand them directly on target + for raw_path in raw_list.split(): + cmd = f"for p in {raw_path}; do if [ -e $p ]; then echo $p; fi; done" + try: + status, output = target.run(cmd) + if status != 0 or not output: + raise Exception() + result += output.split() + except: + bb.warn(f"No file/directory matching path {raw_path}") + + return result + +def retrieve_test_artifacts(target, artifacts_list, target_dir): + import shutil + + local_artifacts_dir = os.path.join(target_dir, "artifacts") + if os.path.isdir(local_artifacts_dir): + shutil.rmtree(local_artifacts_dir) + + os.makedirs(local_artifacts_dir) + for artifact_path in artifacts_list: + if not os.path.isabs(artifact_path): + bb.warn(f"{artifact_path} is not an absolute path") + continue + try: + dest_dir = os.path.join(local_artifacts_dir, os.path.dirname(artifact_path[1:])) + os.makedirs(dest_dir, exist_ok=True) + target.copyFrom(artifact_path, dest_dir) + except: + bb.warn(f"Can not retrieve {artifact_path} from test target") def testimage_main(d): import os @@ -383,6 +425,12 @@ def testimage_main(d): pass results = tc.runTests() complete = True + if results.hasAnyFailingTest(): + artifacts_list = get_artifacts_list(tc.target, d.getVar("TESTIMAGE_FAILED_QA_ARTIFACTS")) + if not artifacts_list: + bb.warn("Could not load artifacts list, skip artifacts retrieval") + else: + retrieve_test_artifacts(tc.target, artifacts_list, get_testimage_json_result_dir(d)) except (KeyboardInterrupt, BlockingIOError) as err: if isinstance(err, KeyboardInterrupt): bb.error('testimage interrupted, shutting down...') From patchwork Fri Jun 9 06:48:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alexis_Lothor=C3=A9?= X-Patchwork-Id: 25315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EBBBC7EE25 for ; Fri, 9 Jun 2023 06:47:57 +0000 (UTC) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by mx.groups.io with SMTP id smtpd.web11.7420.1686293271159675657 for ; Thu, 08 Jun 2023 23:47:51 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=I302UlRX; spf=pass (domain: bootlin.com, ip: 217.70.183.195, mailfrom: alexis.lothore@bootlin.com) X-GND-Sasl: alexis.lothore@bootlin.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1686293269; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3F2gl06ox/sSuq5WQgpKO4sWduQzKIa9KmcJedgpIDI=; b=I302UlRXZla3ire7nbCaxmK38AzJILoFKKApXzu0qYfYDzfZmuAFK1Tb/4bua1NnSXcJp7 IOOmlH98Q+Fqo+7BQa/9hrj71SuLNsRkRWzGhihpS12tADWLa9gn43ayR7id4y2Xp7Z7BA ogHl5efmDQdrks4UXe5J0HECMNvLIoEPoahcos99/qZGem4oMCVrAPujzLkBJjQIM7BHzR F9pBUew4ukwEmuxuc/uLfmGAG973UZ+fxbco9A47Von2uEtBgObNK8xWi5qW+fZtnF0LSw yFUsk6Cv5JDZnWn7yd5Ot0/aGIGr3ulCMyiCHRP39M/wAEUmJm+gO5c888dofw== X-GND-Sasl: alexis.lothore@bootlin.com X-GND-Sasl: alexis.lothore@bootlin.com X-GND-Sasl: alexis.lothore@bootlin.com Received: by mail.gandi.net (Postfix) with ESMTPSA id 84DCD60005; Fri, 9 Jun 2023 06:47:49 +0000 (UTC) From: alexis.lothore@bootlin.com To: Cc: Thomas Petazzoni , Alexandre Belloni , =?utf-8?q?Alexis_Lothor?= =?utf-8?q?=C3=A9?= Subject: [OE-Core][PATCH v3 4/4] core-image-ptest: append ptest directory to artifacts list Date: Fri, 9 Jun 2023 08:48:02 +0200 Message-Id: <20230609064802.11777-5-alexis.lothore@bootlin.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230609064802.11777-1-alexis.lothore@bootlin.com> References: <20230609064802.11777-1-alexis.lothore@bootlin.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 09 Jun 2023 06:47:57 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/182536 From: Alexis Lothoré TESTIMAGE_FAILED_QA_ARTIFACTS is defined in testimage.bbclass with a minimal list of files to retrieve when a test fail. By appending the ptest directory only in core-image-ptest.bb, thanks to multiconfig feature used in the recipe, only failing ptests will lead to corresponding ptest artifacts retrieval, instead of all ptests artifacts retrieval. Signed-off-by: Alexis Lothoré --- meta/recipes-core/images/core-image-ptest.bb | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/recipes-core/images/core-image-ptest.bb b/meta/recipes-core/images/core-image-ptest.bb index 90c26641ba3a..e1be81bb2666 100644 --- a/meta/recipes-core/images/core-image-ptest.bb +++ b/meta/recipes-core/images/core-image-ptest.bb @@ -28,6 +28,7 @@ QB_MEM:virtclass-mcextend-lttng-tools = "-m 4096" QB_MEM:virtclass-mcextend-python3-cryptography = "-m 4096" TEST_SUITES = "ping ssh parselogs ptest" +TESTIMAGE_FAILED_QA_ARTIFACTS:append=" ${libdir}/${MCNAME}/ptest" # Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places PTEST_EXPECT_FAILURE = "1"