From patchwork Fri Jun 2 09:50:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alexis_Lothor=C3=A9?= X-Patchwork-Id: 25047 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CBA8C7EE2F for ; Fri, 2 Jun 2023 09:50:27 +0000 (UTC) Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [217.70.183.198]) by mx.groups.io with SMTP id smtpd.web10.8883.1685699420078968635 for ; Fri, 02 Jun 2023 02:50:20 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=kCqrodAC; spf=pass (domain: bootlin.com, ip: 217.70.183.198, mailfrom: alexis.lothore@bootlin.com) X-GND-Sasl: alexis.lothore@bootlin.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1685699418; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XakE4986FFZiaonPfKCUcsC8TTf0nqLVbsX7lObtkvc=; b=kCqrodAC8hSFDy8WHoyPylS6BT/5ZeMG7u3rTL1oid6rYdMdkT2lqa4PZN8QibjpTZBxVh PSrH2C+DOv98WrXG13x5Dm1pHhUg3tgqzb2UaN5nhLrJVn4a7QP4Zp7H4xhFtjl+Jep/gd gmOFFN0Y5XklkLi64ZmRbyP+7sTO4bKlM9N2Z4ZalWCNxnLMSf00ZV6uEXpyXQ1mRD/e33 pEXICW+O151ssOP0lOm2LsYXxvgdvrxvi3nAF3NHuFzsm3PnynCvk608/ufhNYR4aKNaNe z15z0grIPX9vtMW7C0LIOxPyGVTb6EUV7jBE0DwT0BjrSuC6KVAAY4wz/HHvYg== X-GND-Sasl: alexis.lothore@bootlin.com X-GND-Sasl: alexis.lothore@bootlin.com Received: by mail.gandi.net (Postfix) with ESMTPSA id 5A889C0002; Fri, 2 Jun 2023 09:50:18 +0000 (UTC) From: =?utf-8?q?Alexis_Lothor=C3=A9?= To: Cc: Thomas Petazzoni , Alexandre Belloni Subject: [OE-Core][RFC][PATCH 1/3] oeqa/target/ssh: update options for SCP Date: Fri, 2 Jun 2023 11:50:35 +0200 Message-Id: <20230602095037.97981-2-alexis.lothore@bootlin.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230602095037.97981-1-alexis.lothore@bootlin.com> References: <20230602095037.97981-1-alexis.lothore@bootlin.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 02 Jun 2023 09:50:27 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/182298 Add two options for SCP: - by default scp expects sftp server to be available on target, which is not always true (e.g: core-image-minimal image). Pass -O to force SCP to fallback to legacy SCP protocol - by default scp expects files. Passing -r option allows to copy directories too Signed-off-by: Alexis Lothoré --- meta/lib/oeqa/core/target/ssh.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/meta/lib/oeqa/core/target/ssh.py b/meta/lib/oeqa/core/target/ssh.py index 51079075b5bd..dc17ceecd74d 100644 --- a/meta/lib/oeqa/core/target/ssh.py +++ b/meta/lib/oeqa/core/target/ssh.py @@ -40,8 +40,12 @@ class OESSHTarget(OETarget): '-o', 'StrictHostKeyChecking=no', '-o', 'LogLevel=ERROR' ] + scp_options = [ + '-O', # Allow legacy option for images not containing SFTP server + '-r' + ] self.ssh = ['ssh', '-l', self.user ] + ssh_options - self.scp = ['scp'] + ssh_options + self.scp = ['scp'] + ssh_options + scp_options if port: self.ssh = self.ssh + [ '-p', port ] self.scp = self.scp + [ '-P', port ] From patchwork Fri Jun 2 09:50:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alexis_Lothor=C3=A9?= X-Patchwork-Id: 25046 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68552C7EE24 for ; Fri, 2 Jun 2023 09:50:27 +0000 (UTC) Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [217.70.183.198]) by mx.groups.io with SMTP id smtpd.web11.8830.1685699421010756587 for ; Fri, 02 Jun 2023 02:50:21 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=SguQEBf4; spf=pass (domain: bootlin.com, ip: 217.70.183.198, mailfrom: alexis.lothore@bootlin.com) X-GND-Sasl: alexis.lothore@bootlin.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1685699419; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VhKIhJfqtwOM8Y+AejSNXNUK81sjjOqIBdPMZkBilI0=; b=SguQEBf4uElM5GpLC8ABNTBhJGWsjEX4iGdiz2QQP9GR5Cvfl2CLwA0DNDccIuQ8zBS9C0 YwaDxUQHENOm+E61my42q3p3CZ80BuMokRsHAW8m103dyQiI4BPSrj3Ntlm+5Si78eRnYw 0OmojWa1lVDOTyAauCqrC9b5DRg2XudYNid091G9ePEVcuEFnrQNGpwYxdyiKYOMVadJDG WfQygOQFR2gStzPJ+0iGeUL22sGjRLtyDbuIUrJ6HlHk/iU8/G5AJlurqx6hsFDlqyHuU1 3Qvc8ArPHOXYZhoYLh2zQ+ePxqqPtuL/2snWZWxDQMyP4QyH4oSjDvCvVhpQfg== X-GND-Sasl: alexis.lothore@bootlin.com X-GND-Sasl: alexis.lothore@bootlin.com Received: by mail.gandi.net (Postfix) with ESMTPSA id B6007C0006; Fri, 2 Jun 2023 09:50:18 +0000 (UTC) From: =?utf-8?q?Alexis_Lothor=C3=A9?= To: Cc: Thomas Petazzoni , Alexandre Belloni Subject: [OE-Core][RFC][PATCH 2/3] testimage: shut down DUT later Date: Fri, 2 Jun 2023 11:50:36 +0200 Message-Id: <20230602095037.97981-3-alexis.lothore@bootlin.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230602095037.97981-1-alexis.lothore@bootlin.com> References: <20230602095037.97981-1-alexis.lothore@bootlin.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 02 Jun 2023 09:50:27 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/182301 Move target stop (for Qemu targets: qemu image stop) later in testimage main function, especially _after_ having access to general test results, to allow executing some other actions (like retrieving some files) on DUT depending on test results (e.g. : retrieve some log files) Signed-off-by: Alexis Lothoré --- meta/classes-recipe/testimage.bbclass | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/meta/classes-recipe/testimage.bbclass b/meta/classes-recipe/testimage.bbclass index 1bf0cb450ce4..6b10c1db09f9 100644 --- a/meta/classes-recipe/testimage.bbclass +++ b/meta/classes-recipe/testimage.bbclass @@ -393,7 +393,6 @@ def testimage_main(d): results = tc.results finally: signal.signal(signal.SIGTERM, orig_sigterm_handler) - tc.target.stop() # Show results (if we have them) if results: @@ -404,6 +403,8 @@ def testimage_main(d): dump_streams=d.getVar('TESTREPORT_FULLLOGS')) results.logSummary(pn) + tc.target.stop() + # Copy additional logs to tmp/log/oeqa so it's easier to find them targetdir = os.path.join(get_testimage_json_result_dir(d), d.getVar("PN")) os.makedirs(targetdir, exist_ok=True) @@ -415,6 +416,7 @@ def testimage_main(d): if not results.wasSuccessful(): bb.fatal('%s - FAILED - also check the logs in %s' % (pn, d.getVar("LOG_DIR")), forcelog=True) + def get_runtime_paths(d): """ Returns a list of paths where runtime test must reside. From patchwork Fri Jun 2 09:50:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alexis_Lothor=C3=A9?= X-Patchwork-Id: 25045 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BE4FC7EE2E for ; Fri, 2 Jun 2023 09:50:27 +0000 (UTC) Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [217.70.183.198]) by mx.groups.io with SMTP id smtpd.web10.8885.1685699420725604369 for ; Fri, 02 Jun 2023 02:50:21 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=SjVqcfhR; spf=pass (domain: bootlin.com, ip: 217.70.183.198, mailfrom: alexis.lothore@bootlin.com) X-GND-Sasl: alexis.lothore@bootlin.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1685699419; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=O4I0hO2nZdkAGDGW9kZppHciRxm3gHpa3loYxv8EleY=; b=SjVqcfhRI9cHB3QdEJZ6A2r0+VNL2mTPg7vSqezrwBj5uPJxXMcJFEPL+lEjEjwmQHzwZA FnBeTgvrErNT4lvGkz9Q4pUE2agj9yBY8HI1wYjF9H23vZp4oyX6JstuILTNxd51PyArNn g66nFOB+ZWv0ZEpgLuxYuXLNLaggqyDT2zVtPtZYZRRIlfMJSogLz1BLRFMUHO5MPVT9S1 relH4n/+EXCRgBrxXAWiRGu19XGL8I0qWyJRSQ0wm0j3s7X1rd33tpIhlKANuiTEMGdtrX SbBYerIqPc7OOhs24QhTIHmAr/nh9zXjodLAr4gcgrZwRBqr1mJdZfPvH/11GQ== X-GND-Sasl: alexis.lothore@bootlin.com X-GND-Sasl: alexis.lothore@bootlin.com Received: by mail.gandi.net (Postfix) with ESMTPSA id 2035BC0012; Fri, 2 Jun 2023 09:50:19 +0000 (UTC) From: =?utf-8?q?Alexis_Lothor=C3=A9?= To: Cc: Thomas Petazzoni , Alexandre Belloni Subject: [OE-Core][RFC][PATCH 3/3] testimage: implement test artifacts retriever for failing tests Date: Fri, 2 Jun 2023 11:50:37 +0200 Message-Id: <20230602095037.97981-4-alexis.lothore@bootlin.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230602095037.97981-1-alexis.lothore@bootlin.com> References: <20230602095037.97981-1-alexis.lothore@bootlin.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 02 Jun 2023 09:50:27 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/182300 Add a basic artifacts retrievers in testimage class which: - triggers when at least one runtime test fails - reads a list of files to retrieve from ARTIFACTS_LIST_PATH - retrieve those files over scp thanks to existing ssh class - store those files in an "artifacts" directory in "tmp/log/oeqa/" Bring partial solution to [YOCTO #14901] Signed-off-by: Alexis Lothoré --- meta/classes-recipe/testimage.bbclass | 51 +++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/meta/classes-recipe/testimage.bbclass b/meta/classes-recipe/testimage.bbclass index 6b10c1db09f9..3fd3dd5b0264 100644 --- a/meta/classes-recipe/testimage.bbclass +++ b/meta/classes-recipe/testimage.bbclass @@ -18,6 +18,12 @@ inherit image-artifact-names TESTIMAGE_AUTO ??= "0" +# When any test fails, if path to an artifacts configuration (listing +# files/directories to retrieve on target) has been provided, all listed +# elements will be downloaded from target before shutting it down + +ARTIFACTS_LIST_PATH ??= "" + # You can set (or append to) TEST_SUITES in local.conf to select the tests # which you want to run for your target. # The test names are the module names in meta/lib/oeqa/runtime/cases. @@ -192,6 +198,45 @@ def get_testimage_boot_patterns(d): boot_patterns[flag] = flagval.encode().decode('unicode-escape') return boot_patterns +def load_artifacts_list(artifacts_conf_path): + import json + + if not artifacts_conf_path: + return None + + if not os.path.isfile(artifacts_conf_path): + return None + + try: + with open(artifacts_conf_path) as f: + artifacts_list = json.load(f) + except json.decoder.JSONDecodeError as e: + bb.warn(f"Invalid tests artifact list format: {e.lineno}:{e.colno} : {e.msg}") + return None + + if 'artifacts' in artifacts_list: + return artifacts_list['artifacts'] + + return None + +def retrieve_test_artifacts(target, artifacts_list, target_dir): + import shutil + + local_artifacts_dir = os.path.join(target_dir, "artifacts") + if os.path.isdir(local_artifacts_dir): + shutil.rmtree(local_artifacts_dir) + + os.makedirs(local_artifacts_dir) + for artifact_path in artifacts_list: + if not os.path.isabs(artifact_path): + bb.warn(f"{artifact_path} is not an absolute path") + continue + try: + dest_dir = os.path.join(local_artifacts_dir, os.path.dirname(artifact_path[1:])) + os.makedirs(dest_dir, exist_ok=True) + target.copyFrom(artifact_path, dest_dir) + except: + bb.warn(f"Can not retrieve {artifact_path} from test target") def testimage_main(d): import os @@ -402,6 +447,12 @@ def testimage_main(d): get_testimage_result_id(configuration), dump_streams=d.getVar('TESTREPORT_FULLLOGS')) results.logSummary(pn) + if not results.wasSuccessful(): + artifacts_list = load_artifacts_list(d.getVar("ARTIFACTS_LIST_PATH")) + if not artifacts_list: + bb.warn("Could not load artifacts list, skip artifacts retrieval") + else: + retrieve_test_artifacts(tc.target, artifacts_list, get_testimage_json_result_dir(d)) tc.target.stop()