From patchwork Mon Jul 17 18:56:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Hoyes X-Patchwork-Id: 27536 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8198DEB64DC for ; Mon, 17 Jul 2023 18:56:51 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.3236.1689620209964076757 for ; Mon, 17 Jul 2023 11:56:50 -0700 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: peter.hoyes@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 92EB4C15; Mon, 17 Jul 2023 11:57:32 -0700 (PDT) Received: from e125920.arm.com (unknown [10.57.87.125]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 82E853F67D; Mon, 17 Jul 2023 11:56:48 -0700 (PDT) From: Peter Hoyes To: meta-arm@lists.yoctoproject.org Cc: Peter Hoyes Subject: [PATCH 1/5] runfvp: Add missing conffile include Date: Mon, 17 Jul 2023 19:56:22 +0100 Message-Id: <20230717185626.1793017-1-peter.hoyes@arm.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 17 Jul 2023 18:56:51 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/meta-arm/message/4880 From: Peter Hoyes Using runfvp without explicitly specifying the fvpconf path currently fails due to a missing fvp.conffile include. Signed-off-by: Peter Hoyes --- scripts/runfvp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/runfvp b/scripts/runfvp index c2e536c8..0ca3a1b5 100755 --- a/scripts/runfvp +++ b/scripts/runfvp @@ -14,7 +14,7 @@ logger = logging.getLogger("RunFVP") libdir = pathlib.Path(__file__).parents[1] / "meta-arm" / "lib" sys.path.insert(0, str(libdir)) -from fvp import terminal, runner +from fvp import conffile, terminal, runner def parse_args(arguments): import argparse From patchwork Mon Jul 17 18:56:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Hoyes X-Patchwork-Id: 27539 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84AA1C3DA41 for ; Mon, 17 Jul 2023 18:57:01 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.3323.1689620216779231487 for ; Mon, 17 Jul 2023 11:56:57 -0700 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: peter.hoyes@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2D857C15; Mon, 17 Jul 2023 11:57:39 -0700 (PDT) Received: from e125920.arm.com (unknown [10.57.87.125]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id F304F3F67D; Mon, 17 Jul 2023 11:56:54 -0700 (PDT) From: Peter Hoyes To: meta-arm@lists.yoctoproject.org Cc: Peter Hoyes Subject: [PATCH 2/5] arm/oeqa: Merge all OEFVP*Target classes Date: Mon, 17 Jul 2023 19:56:23 +0100 Message-Id: <20230717185626.1793017-2-peter.hoyes@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230717185626.1793017-1-peter.hoyes@arm.com> References: <20230717185626.1793017-1-peter.hoyes@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 17 Jul 2023 18:57:01 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/meta-arm/message/4881 From: Peter Hoyes The differences between OEFVPTarget, OEFVPSSHTarget and OEFVPSerialTarget are not obvious and there is a lot of duplication. Merge all the logic into one OEFVPTarget (again). This has the following features: * Run SSH commands * Run serial console assertions * Lazily await a Linux login prompt while running test cases (only when self.target.run is called). Signed-off-by: Peter Hoyes --- documentation/oeqa-fvp.md | 46 +++---- .../conf/machine/corstone1000-fvp.conf | 2 +- meta-arm-bsp/conf/machine/corstone500.conf | 2 +- .../conf/machine/fvp-baser-aemv8r64.conf | 2 +- meta-arm-bsp/conf/machine/tc1.conf | 2 +- meta-arm/lib/oeqa/controllers/fvp.py | 118 +++++++----------- meta-arm/lib/oeqa/runtime/cases/linuxboot.py | 4 +- 7 files changed, 71 insertions(+), 105 deletions(-) diff --git a/documentation/oeqa-fvp.md b/documentation/oeqa-fvp.md index e1468851..b39e0963 100644 --- a/documentation/oeqa-fvp.md +++ b/documentation/oeqa-fvp.md @@ -4,32 +4,36 @@ OE-Core's [oeqa][OEQA] framework provides a method of performing runtime tests o Tests can be configured to run automatically post-build by setting the variable `TESTIMAGE_AUTO="1"`, e.g. in your Kas file or local.conf. -There are two main methods of testing, using different test "targets". Both test targets generate an additional log file with the prefix 'fvp_log' in the image recipe's `${WORKDIR}/testimage` containing the FVP's stdout. +meta-arm provides the OEFVPTarget which must be set up in the machine configuration: +``` +TEST_TARGET = "OEFVPTarget" +TEST_SERVER_IP = "127.0.0.1" +TEST_TARGET_IP = "127.0.0.1:8022" +IMAGE_FEATURES:append = " ssh-server-dropbear" +FVP_CONFIG[bp.virtio_net.hostbridge.userNetPorts] ?= "8022=22" +FVP_CONSOLES[default] = "terminal_0" +FVP_CONSOLES[tf-a] = "s_terminal_0" +``` + +The test target also generates a log file with the prefix 'fvp_log' in the image recipe's `${WORKDIR}/testimage` containing the FVP's stdout. -## OEFVPTarget +OEFVPTarget supports two different test interfaces - SSH and pexpect. -This runs test cases on a machine using SSH. It therefore requires that an SSH server is installed in the image. +## SSH -In test cases, the primary interface with the target is, e.g: +As in OEQA in OE-core, tests cases can run commands on the machine using SSH. It therefore requires that an SSH server is installed in the image. + +This uses the `run` method on the target, e.g: ``` (status, output) = self.target.run('uname -a') ``` -which runs a single command on the target (using `ssh -c`) and returns the status code and the output. It is therefore useful for running tests in a Linux environment. +which executes a single command on the target (using `ssh -c`) and returns the status code and the output. It is therefore useful for running tests in a Linux environment. For examples of test cases, see meta/lib/oeqa/runtime/cases in OE-Core. The majority of test cases depend on `ssh.SSHTest.test_ssh`, which first validates that the SSH connection is functioning. -Example machine configuration: -``` -TEST_TARGET = "OEFVPTarget" -TEST_SERVER_IP = "127.0.0.1" -TEST_TARGET_IP = "127.0.0.1:8022" -IMAGE_FEATURES:append = " ssh-server-dropbear" -FVP_CONFIG[bp.virtio_net.hostbridge.userNetPorts] ?= "8022=22" -``` - -## OEFVPSerialTarget +## pexpect -This runs tests against one or more serial consoles on the FVP. It is more flexible than OEFVPTarget, but test cases written for this test target do not support the test cases in OE-core. As it does not require an SSH server, it is suitable for machines with performance or memory limitations. +To support firmware and baremetal testing, OEFVPTarget also allows test cases to make assertions against one or more consoles using the pexpect library. Internally, this test target launches a [Pexpect][PEXPECT] instance for each entry in FVP_CONSOLES which can be used with the provided alias. The whole Pexpect API is exposed on the target, where the alias is always passed as the first argument, e.g.: ``` @@ -39,16 +43,6 @@ self.assertNotIn(b'ERROR:', self.target.before('tf-a')) For an example of a full test case, see meta-arm/lib/oeqa/runtime/cases/linuxboot.py This test case can be used to minimally verify that a machine boots to a Linux shell. The default timeout is 10 minutes, but this can be configured with the variable TEST_FVP_LINUX_BOOT_TIMEOUT, which expects a value in seconds. -The SSH interface described above is also available on OEFVPSerialTarget to support writing a set of hybrid test suites that use a combination of serial and SSH access. Note however that this test target does not guarantee that Linux has booted to shell prior to running any tests, so the test cases in OE-core are not supported. - -Example machine configuration: -``` -TEST_TARGET="OEFVPSerialTarget" -TEST_SUITES="linuxboot" -FVP_CONSOLES[default] = "terminal_0" -FVP_CONSOLES[tf-a] = "s_terminal_0" -``` - [OEQA]: https://docs.yoctoproject.org/test-manual/intro.html [FVP]: https://developer.arm.com/tools-and-software/simulation-models/fixed-virtual-platforms [RUNFVP]: runfvp.md diff --git a/meta-arm-bsp/conf/machine/corstone1000-fvp.conf b/meta-arm-bsp/conf/machine/corstone1000-fvp.conf index 66236515..40f69297 100644 --- a/meta-arm-bsp/conf/machine/corstone1000-fvp.conf +++ b/meta-arm-bsp/conf/machine/corstone1000-fvp.conf @@ -8,7 +8,7 @@ TFA_TARGET_PLATFORM = "fvp" TFM_PLATFORM_IS_FVP = "TRUE" # testimage config -TEST_TARGET = "OEFVPSerialTarget" +TEST_TARGET = "OEFVPTarget" TEST_SUITES = "linuxboot" # FVP Config diff --git a/meta-arm-bsp/conf/machine/corstone500.conf b/meta-arm-bsp/conf/machine/corstone500.conf index 4794028a..36ba7703 100644 --- a/meta-arm-bsp/conf/machine/corstone500.conf +++ b/meta-arm-bsp/conf/machine/corstone500.conf @@ -32,7 +32,7 @@ WKS_FILE_DEPENDS:append = " ${EXTRA_IMAGEDEPENDS}" WKS_FILE ?= "core-image-minimal.corstone500.wks" -TEST_TARGET = "OEFVPSerialTarget" +TEST_TARGET = "OEFVPTarget" TEST_SUITES = "linuxboot" FVP_PROVIDER ?= "fvp-corstone500-native" diff --git a/meta-arm-bsp/conf/machine/fvp-baser-aemv8r64.conf b/meta-arm-bsp/conf/machine/fvp-baser-aemv8r64.conf index 62c9cbd0..b8a6aa46 100644 --- a/meta-arm-bsp/conf/machine/fvp-baser-aemv8r64.conf +++ b/meta-arm-bsp/conf/machine/fvp-baser-aemv8r64.conf @@ -29,7 +29,7 @@ MACHINE_FEATURES:append = " efi" MACHINE_EXTRA_RRECOMMENDS += "ssh-pregen-hostkeys" # testimage configuration -TEST_TARGET = "OEFVPSerialTarget" +TEST_TARGET = "OEFVPTarget" TEST_SUITES = "linuxboot" TEST_TARGET_IP ?= "127.0.0.1:8022" TEST_SERVER_IP ?= "127.0.1.1" diff --git a/meta-arm-bsp/conf/machine/tc1.conf b/meta-arm-bsp/conf/machine/tc1.conf index 5f68cc7a..bba2c191 100644 --- a/meta-arm-bsp/conf/machine/tc1.conf +++ b/meta-arm-bsp/conf/machine/tc1.conf @@ -6,7 +6,7 @@ require conf/machine/include/tc.inc -TEST_TARGET = "OEFVPSerialTarget" +TEST_TARGET = "OEFVPTarget" TEST_SUITES = "linuxboot" # FVP Config diff --git a/meta-arm/lib/oeqa/controllers/fvp.py b/meta-arm/lib/oeqa/controllers/fvp.py index 38484072..ea572dd7 100644 --- a/meta-arm/lib/oeqa/controllers/fvp.py +++ b/meta-arm/lib/oeqa/controllers/fvp.py @@ -6,11 +6,14 @@ from oeqa.core.target.ssh import OESSHTarget from fvp import runner -class OEFVPSSHTarget(OESSHTarget): +class OEFVPTarget(OESSHTarget): """ - Base class for meta-arm FVP targets. - Contains common logic to start and stop an FVP. + For compatibility with OE-core test cases, this target's start() method + waits for a Linux shell before returning to ensure that SSH commands work + with the default test dependencies. """ + DEFAULT_CONSOLE = "default" + def __init__(self, logger, target_ip, server_ip, timeout=300, user='root', port=None, dir_image=None, rootfs=None, bootlog=None, **kwargs): super().__init__(logger, target_ip, server_ip, timeout, user, port) @@ -19,90 +22,45 @@ class OEFVPSSHTarget(OESSHTarget): basename = pathlib.Path(rootfs) basename = basename.name.replace("".join(basename.suffixes), "") self.fvpconf = image_dir / (basename + ".fvpconf") - self.bootlog = bootlog - if not self.fvpconf.exists(): raise FileNotFoundError(f"Cannot find {self.fvpconf}") - def _after_start(self): - pass + self.bootlog = bootlog + self.terminals = {} + self.booted = False def start(self, **kwargs): self.fvp_log = self._create_logfile("fvp") self.fvp = runner.FVPRunner(self.logger) self.fvp.start(self.fvpconf, stdout=self.fvp_log) self.logger.debug(f"Started FVP PID {self.fvp.pid()}") - self._after_start() - - def stop(self, **kwargs): - returncode = self.fvp.stop() - self.logger.debug(f"Stopped FVP with return code {returncode}") + self._setup_consoles() - def _create_logfile(self, name): - if not self.bootlog: - return None - - test_log_path = pathlib.Path(self.bootlog).parent - test_log_suffix = pathlib.Path(self.bootlog).suffix - fvp_log_file = f"{name}_log{test_log_suffix}" - fvp_log_path = pathlib.Path(test_log_path, fvp_log_file) - fvp_log_symlink = pathlib.Path(test_log_path, f"{name}_log") - try: - os.remove(fvp_log_symlink) - except: - pass - os.symlink(fvp_log_file, fvp_log_symlink) - return open(fvp_log_path, 'wb') - - -class OEFVPTarget(OEFVPSSHTarget): - """ - For compatibility with OE-core test cases, this target's start() method - waits for a Linux shell before returning to ensure that SSH commands work - with the default test dependencies. - """ - def __init__(self, logger, target_ip, server_ip, **kwargs): - super().__init__(logger, target_ip, server_ip, **kwargs) - self.logfile = self.bootlog and open(self.bootlog, "wb") or None + def await_boot(self): + if self.booted: + return # FVPs boot slowly, so allow ten minutes - self.boot_timeout = 10 * 60 + boot_timeout = 10*60 + try: + self.expect(OEFVPTarget.DEFAULT_CONSOLE, "login\\:", timeout=boot_timeout) + self.logger.debug("Found login prompt") + self.booted = True + except pexpect.TIMEOUT: + self.logger.info("Timed out waiting for login prompt.") + self.logger.info("Boot log follows:") + self.logger.info(b"\n".join(self.before(OEFVPTarget.DEFAULT_CONSOLE).splitlines()[-200:]).decode("utf-8", errors="replace")) + raise RuntimeError("Failed to start FVP.") - def _after_start(self): - with open(self.fvp_log.name, 'rb') as logfile: - parser = runner.ConsolePortParser(logfile) - config = self.fvp.getConfig() - self.logger.debug(f"Awaiting console on terminal {config['consoles']['default']}") - port = parser.parse_port(config['consoles']['default']) - console = self.fvp.create_pexpect(port) - try: - console.expect("login\\:", timeout=self.boot_timeout) - self.logger.debug("Found login prompt") - except pexpect.TIMEOUT: - self.logger.info("Timed out waiting for login prompt.") - self.logger.info("Boot log follows:") - self.logger.info(b"\n".join(console.before.splitlines()[-200:]).decode("utf-8", errors="replace")) - raise RuntimeError("Failed to start FVP.") - - -class OEFVPSerialTarget(OEFVPSSHTarget): - """ - This target is intended for interaction with the target over one or more - telnet consoles using pexpect. - - This still depends on OEFVPSSHTarget so SSH commands can still be run on - the target, but note that this class does not inherently guarantee that - the SSH server is running prior to running test cases. Test cases that use - SSH should first validate that SSH is available, e.g. by depending on the - "linuxboot" test case in meta-arm. - """ - DEFAULT_CONSOLE = "default" + def stop(self, **kwargs): + returncode = self.fvp.stop() + self.logger.debug(f"Stopped FVP with return code {returncode}") - def __init__(self, logger, target_ip, server_ip, **kwargs): - super().__init__(logger, target_ip, server_ip, **kwargs) - self.terminals = {} + def run(self, cmd, timeout=None): + self.await_boot() + return super().run(cmd, timeout) - def _after_start(self): + def _setup_consoles(self): with open(self.fvp_log.name, 'rb') as logfile: parser = runner.ConsolePortParser(logfile) config = self.fvp.getConfig() @@ -119,6 +77,22 @@ class OEFVPSerialTarget(OEFVPSSHTarget): default_test_file = f"{name}_log{self.test_log_suffix}" os.symlink(default_test_file, self.bootlog) + def _create_logfile(self, name): + if not self.bootlog: + return None + + test_log_path = pathlib.Path(self.bootlog).parent + test_log_suffix = pathlib.Path(self.bootlog).suffix + fvp_log_file = f"{name}_log{test_log_suffix}" + fvp_log_path = pathlib.Path(test_log_path, fvp_log_file) + fvp_log_symlink = pathlib.Path(test_log_path, f"{name}_log") + try: + os.remove(fvp_log_symlink) + except: + pass + os.symlink(fvp_log_file, fvp_log_symlink) + return open(fvp_log_path, 'wb') + def _get_terminal(self, name): return self.terminals[name] diff --git a/meta-arm/lib/oeqa/runtime/cases/linuxboot.py b/meta-arm/lib/oeqa/runtime/cases/linuxboot.py index 99a8e78b..184eb6c9 100644 --- a/meta-arm/lib/oeqa/runtime/cases/linuxboot.py +++ b/meta-arm/lib/oeqa/runtime/cases/linuxboot.py @@ -5,9 +5,7 @@ from oeqa.runtime.case import OERuntimeTestCase class LinuxBootTest(OERuntimeTestCase): """ - This test case is only compatible with the OEFVPSerialTarget as it uses - the pexpect interface. It waits for a Linux login prompt on the default - console. + This test waits for a Linux login prompt on the default console. """ def setUp(self): From patchwork Mon Jul 17 18:56:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Hoyes X-Patchwork-Id: 27538 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF5FDC001B0 for ; Mon, 17 Jul 2023 18:57:01 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.3240.1689620217519336043 for ; Mon, 17 Jul 2023 11:56:57 -0700 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: peter.hoyes@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8EB62D75; Mon, 17 Jul 2023 11:57:40 -0700 (PDT) Received: from e125920.arm.com (unknown [10.57.87.125]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 7E5133F67D; Mon, 17 Jul 2023 11:56:56 -0700 (PDT) From: Peter Hoyes To: meta-arm@lists.yoctoproject.org Cc: Peter Hoyes Subject: [PATCH 3/5] arm/OEFVPTarget: Add support for model state transitions Date: Mon, 17 Jul 2023 19:56:24 +0100 Message-Id: <20230717185626.1793017-3-peter.hoyes@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230717185626.1793017-1-peter.hoyes@arm.com> References: <20230717185626.1793017-1-peter.hoyes@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 17 Jul 2023 18:57:01 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/meta-arm/message/4882 From: Peter Hoyes To better support firmware testing alongside Linux runtime testing, introduce model state support to OEFVPTarget. The following states are supported using self.target.transition(state): * off * on * linux Instead of assuming a specific state in OEFVPTarget.start, responsibility is delegated to test cases to lazily put the model in the required state. But to support OE-core test cases, OEFVPTarget.run automatically puts the model in the "linux" state for running the command. Firmware and Linux tests can subsequently run alongside each other without introducing complex test dependencies. The concept is inspired by Labgrid strategies [1], albeit simplified. Tweak log file handling so that output is collected across (possibly) multiple model processes. [1] https://labgrid.readthedocs.io/en/latest/overview.html#strategies Signed-off-by: Peter Hoyes --- meta-arm/lib/oeqa/controllers/fvp.py | 77 ++++++++++++++++++---------- 1 file changed, 49 insertions(+), 28 deletions(-) diff --git a/meta-arm/lib/oeqa/controllers/fvp.py b/meta-arm/lib/oeqa/controllers/fvp.py index ea572dd7..cfd8c4e9 100644 --- a/meta-arm/lib/oeqa/controllers/fvp.py +++ b/meta-arm/lib/oeqa/controllers/fvp.py @@ -1,3 +1,5 @@ +import contextlib +import enum import pathlib import pexpect import os @@ -5,6 +7,11 @@ import os from oeqa.core.target.ssh import OESSHTarget from fvp import runner +class OEFVPTargetState(str, enum.Enum): + OFF = "off" + ON = "on" + LINUX = "linux" + class OEFVPTarget(OESSHTarget): """ @@ -27,37 +34,50 @@ class OEFVPTarget(OESSHTarget): self.bootlog = bootlog self.terminals = {} - self.booted = False + self.stack = None + self.state = OEFVPTargetState.OFF - def start(self, **kwargs): - self.fvp_log = self._create_logfile("fvp") - self.fvp = runner.FVPRunner(self.logger) - self.fvp.start(self.fvpconf, stdout=self.fvp_log) - self.logger.debug(f"Started FVP PID {self.fvp.pid()}") - self._setup_consoles() - - def await_boot(self): - if self.booted: + def transition(self, state, timeout=10*60): + if state == self.state: return - # FVPs boot slowly, so allow ten minutes - boot_timeout = 10*60 - try: - self.expect(OEFVPTarget.DEFAULT_CONSOLE, "login\\:", timeout=boot_timeout) - self.logger.debug("Found login prompt") - self.booted = True - except pexpect.TIMEOUT: - self.logger.info("Timed out waiting for login prompt.") - self.logger.info("Boot log follows:") - self.logger.info(b"\n".join(self.before(OEFVPTarget.DEFAULT_CONSOLE).splitlines()[-200:]).decode("utf-8", errors="replace")) - raise RuntimeError("Failed to start FVP.") + if state == OEFVPTargetState.OFF: + returncode = self.fvp.stop() + self.logger.debug(f"Stopped FVP with return code {returncode}") + self.stack.close() + elif state == OEFVPTargetState.ON: + self.transition(OEFVPTargetState.OFF, timeout) + self.stack = contextlib.ExitStack() + self.fvp = runner.FVPRunner(self.logger) + self.fvp_log = self._create_logfile("fvp", "wb") + self.fvp.start(self.fvpconf, stdout=self.fvp_log) + self.logger.debug(f"Started FVP PID {self.fvp.pid()}") + self._setup_consoles() + elif state == OEFVPTargetState.LINUX: + self.transition(OEFVPTargetState.ON, timeout) + try: + self.expect(OEFVPTarget.DEFAULT_CONSOLE, "login\\:", timeout=timeout) + self.logger.debug("Found login prompt") + self.state = OEFVPTargetState.LINUX + except pexpect.TIMEOUT: + self.logger.info("Timed out waiting for login prompt.") + self.logger.info("Boot log follows:") + self.logger.info(b"\n".join(self.before(OEFVPTarget.DEFAULT_CONSOLE).splitlines()[-200:]).decode("utf-8", errors="replace")) + raise RuntimeError("Failed to start FVP.") + + self.logger.info(f"Transitioned to {state}") + self.state = state + + def start(self, **kwargs): + # No-op - put the FVP in the required state lazily + pass def stop(self, **kwargs): - returncode = self.fvp.stop() - self.logger.debug(f"Stopped FVP with return code {returncode}") + self.transition(OEFVPTargetState.OFF) def run(self, cmd, timeout=None): - self.await_boot() + # Running a command implies the LINUX state + self.transition(OEFVPTargetState.LINUX) return super().run(cmd, timeout) def _setup_consoles(self): @@ -73,11 +93,12 @@ class OEFVPTarget(OESSHTarget): # testimage.bbclass expects to see a log file at `bootlog`, # so make a symlink to the 'default' log file - if name == 'default': - default_test_file = f"{name}_log{self.test_log_suffix}" + test_log_suffix = pathlib.Path(self.bootlog).suffix + default_test_file = f"{name}_log{test_log_suffix}" + if name == 'default' and not os.path.exists(self.bootlog): os.symlink(default_test_file, self.bootlog) - def _create_logfile(self, name): + def _create_logfile(self, name, mode='ab'): if not self.bootlog: return None @@ -91,7 +112,7 @@ class OEFVPTarget(OESSHTarget): except: pass os.symlink(fvp_log_file, fvp_log_symlink) - return open(fvp_log_path, 'wb') + return self.stack.enter_context(open(fvp_log_path, mode)) def _get_terminal(self, name): return self.terminals[name] From patchwork Mon Jul 17 18:56:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Hoyes X-Patchwork-Id: 27540 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82131C0015E for ; Mon, 17 Jul 2023 18:57:01 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.3242.1689620219135449322 for ; Mon, 17 Jul 2023 11:56:59 -0700 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: peter.hoyes@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 31E7CC15; Mon, 17 Jul 2023 11:57:42 -0700 (PDT) Received: from e125920.arm.com (unknown [10.57.87.125]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0EF383F67D; Mon, 17 Jul 2023 11:56:57 -0700 (PDT) From: Peter Hoyes To: meta-arm@lists.yoctoproject.org Cc: Peter Hoyes Subject: [PATCH 4/5] arm/oeqa: Convert linuxboot test case into fvp_boot Date: Mon, 17 Jul 2023 19:56:25 +0100 Message-Id: <20230717185626.1793017-4-peter.hoyes@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230717185626.1793017-1-peter.hoyes@arm.com> References: <20230717185626.1793017-1-peter.hoyes@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 17 Jul 2023 18:57:01 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/meta-arm/message/4883 From: Peter Hoyes The linuxboot test case is already FVP-specific due to the use of the OEFVPTarget pexpect interface. Clarify this by renaming to fvp_boot. So that fvp_boot can be used alongside other OEQA test cases (e.g. those in OE-core): * Call self.target.transition("off") at the start of the test to ensure the model starts from reset * Call self.target.transition("linux") to reuse the "wait for boot" logic in OEFVPTarget. Additionally, minimally validate the firmware boot by checking for common error patterns in all console logs. Expose the runfvp config in OEFVPTarget to support this. Align the list of test cases executed on both fvp-base and fvp-baser-aemv8r64 by using TEST_CASES:append = " fvp_boot" for both. Signed-off-by: Peter Hoyes --- .../conf/machine/corstone1000-fvp.conf | 2 +- meta-arm-bsp/conf/machine/corstone500.conf | 2 +- .../conf/machine/fvp-baser-aemv8r64.conf | 2 +- .../conf/machine/include/fvp-common.inc | 1 + meta-arm-bsp/conf/machine/tc1.conf | 2 +- meta-arm/lib/oeqa/controllers/fvp.py | 4 +++ meta-arm/lib/oeqa/runtime/cases/fvp_boot.py | 25 +++++++++++++++++++ meta-arm/lib/oeqa/runtime/cases/linuxboot.py | 17 ------------- 8 files changed, 34 insertions(+), 21 deletions(-) create mode 100644 meta-arm/lib/oeqa/runtime/cases/fvp_boot.py delete mode 100644 meta-arm/lib/oeqa/runtime/cases/linuxboot.py diff --git a/meta-arm-bsp/conf/machine/corstone1000-fvp.conf b/meta-arm-bsp/conf/machine/corstone1000-fvp.conf index 40f69297..449f5658 100644 --- a/meta-arm-bsp/conf/machine/corstone1000-fvp.conf +++ b/meta-arm-bsp/conf/machine/corstone1000-fvp.conf @@ -9,7 +9,7 @@ TFM_PLATFORM_IS_FVP = "TRUE" # testimage config TEST_TARGET = "OEFVPTarget" -TEST_SUITES = "linuxboot" +TEST_SUITES = "fvp_boot" # FVP Config FVP_PROVIDER ?= "fvp-corstone1000-native" diff --git a/meta-arm-bsp/conf/machine/corstone500.conf b/meta-arm-bsp/conf/machine/corstone500.conf index 36ba7703..a688ad2f 100644 --- a/meta-arm-bsp/conf/machine/corstone500.conf +++ b/meta-arm-bsp/conf/machine/corstone500.conf @@ -33,7 +33,7 @@ WKS_FILE_DEPENDS:append = " ${EXTRA_IMAGEDEPENDS}" WKS_FILE ?= "core-image-minimal.corstone500.wks" TEST_TARGET = "OEFVPTarget" -TEST_SUITES = "linuxboot" +TEST_SUITES = "fvp_boot" FVP_PROVIDER ?= "fvp-corstone500-native" FVP_EXE ?= "FVP_Corstone-500" diff --git a/meta-arm-bsp/conf/machine/fvp-baser-aemv8r64.conf b/meta-arm-bsp/conf/machine/fvp-baser-aemv8r64.conf index b8a6aa46..7dbc53a1 100644 --- a/meta-arm-bsp/conf/machine/fvp-baser-aemv8r64.conf +++ b/meta-arm-bsp/conf/machine/fvp-baser-aemv8r64.conf @@ -30,7 +30,7 @@ MACHINE_EXTRA_RRECOMMENDS += "ssh-pregen-hostkeys" # testimage configuration TEST_TARGET = "OEFVPTarget" -TEST_SUITES = "linuxboot" +TEST_SUITES:append = " fvp_boot" TEST_TARGET_IP ?= "127.0.0.1:8022" TEST_SERVER_IP ?= "127.0.1.1" diff --git a/meta-arm-bsp/conf/machine/include/fvp-common.inc b/meta-arm-bsp/conf/machine/include/fvp-common.inc index 47b7ffce..f80ac4c2 100644 --- a/meta-arm-bsp/conf/machine/include/fvp-common.inc +++ b/meta-arm-bsp/conf/machine/include/fvp-common.inc @@ -24,6 +24,7 @@ MACHINE_EXTRA_RRECOMMENDS += "ssh-pregen-hostkeys" TEST_TARGET = "OEFVPTarget" TEST_TARGET_IP = "127.0.0.1:8022" +TEST_SUITES:append = " fvp_boot" FVP_PROVIDER ?= "fvp-base-a-aem-native" FVP_EXE ?= "FVP_Base_RevC-2xAEMvA" diff --git a/meta-arm-bsp/conf/machine/tc1.conf b/meta-arm-bsp/conf/machine/tc1.conf index bba2c191..31bcc2fb 100644 --- a/meta-arm-bsp/conf/machine/tc1.conf +++ b/meta-arm-bsp/conf/machine/tc1.conf @@ -7,7 +7,7 @@ require conf/machine/include/tc.inc TEST_TARGET = "OEFVPTarget" -TEST_SUITES = "linuxboot" +TEST_SUITES = "fvp_boot" # FVP Config FVP_PROVIDER ?= "fvp-tc1-native" diff --git a/meta-arm/lib/oeqa/controllers/fvp.py b/meta-arm/lib/oeqa/controllers/fvp.py index cfd8c4e9..80f72aab 100644 --- a/meta-arm/lib/oeqa/controllers/fvp.py +++ b/meta-arm/lib/oeqa/controllers/fvp.py @@ -132,3 +132,7 @@ class OEFVPTarget(OESSHTarget): return attr return call_pexpect + + @property + def config(self): + return self.fvp.getConfig() diff --git a/meta-arm/lib/oeqa/runtime/cases/fvp_boot.py b/meta-arm/lib/oeqa/runtime/cases/fvp_boot.py new file mode 100644 index 00000000..dce52776 --- /dev/null +++ b/meta-arm/lib/oeqa/runtime/cases/fvp_boot.py @@ -0,0 +1,25 @@ +# SPDX-License-Identifier: MIT + +from oeqa.runtime.case import OERuntimeTestCase +import pexpect + + +class FVPBootTest(OERuntimeTestCase): + """ + This test waits for a Linux login prompt on the default console. It is + dependent on the OEFVPTarget test controller + """ + + def test_fvp_boot(self): + self.target.transition("off") + timeout = int(self.td.get('TEST_FVP_LINUX_BOOT_TIMEOUT') or 10*60) + self.target.transition("linux", timeout) + + # Check for common error patterns on all consoles + for console in self.target.config['consoles']: + # "expect" a timeout when searching for the error patterns + match = self.target.expect(console, + [br'(\[ERR\]|\[ERROR\]|ERROR\:)', + pexpect.TIMEOUT], + timeout=0) + self.assertEqual(match, 1) diff --git a/meta-arm/lib/oeqa/runtime/cases/linuxboot.py b/meta-arm/lib/oeqa/runtime/cases/linuxboot.py deleted file mode 100644 index 184eb6c9..00000000 --- a/meta-arm/lib/oeqa/runtime/cases/linuxboot.py +++ /dev/null @@ -1,17 +0,0 @@ -# SPDX-License-Identifier: MIT - -from oeqa.runtime.case import OERuntimeTestCase - - -class LinuxBootTest(OERuntimeTestCase): - """ - This test waits for a Linux login prompt on the default console. - """ - - def setUp(self): - self.console = self.target.DEFAULT_CONSOLE - self.timeout = int(self.td.get('TEST_FVP_LINUX_BOOT_TIMEOUT') or 10*60) - - def test_linux_boot(self): - self.logger.info(f"{self.console}: Waiting for login prompt") - self.target.expect(self.console, r"login\:", self.timeout) From patchwork Mon Jul 17 18:56:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Hoyes X-Patchwork-Id: 27537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 820F2EB64DC for ; Mon, 17 Jul 2023 18:57:01 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.3244.1689620220821407625 for ; Mon, 17 Jul 2023 11:57:00 -0700 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: peter.hoyes@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D80C9C15; Mon, 17 Jul 2023 11:57:43 -0700 (PDT) Received: from e125920.arm.com (unknown [10.57.87.125]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id AA7533F67D; Mon, 17 Jul 2023 11:56:59 -0700 (PDT) From: Peter Hoyes To: meta-arm@lists.yoctoproject.org Cc: Peter Hoyes Subject: [PATCH 5/5] arm/oeqa: Introduce the fvp_devices test suite Date: Mon, 17 Jul 2023 19:56:26 +0100 Message-Id: <20230717185626.1793017-5-peter.hoyes@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230717185626.1793017-1-peter.hoyes@arm.com> References: <20230717185626.1793017-1-peter.hoyes@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 17 Jul 2023 18:57:01 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/meta-arm/message/4884 From: Peter Hoyes The fvp_devices test suite can be used to verify the following functionality at runtime, common to most FVPs: * CPU hotplug * virtio-net device presence and functionality * virtio-rng device presence and functionality * PL031 RTC device presence and functionality * SP805 watchdog device presence The list of devices to be tested can be configured by a BSP using the variable TEST_FVP_DEVICES. Add this test suite for fvp-base and fvp-baser-aemv8r64. Signed-off-by: Peter Hoyes --- .../conf/machine/fvp-baser-aemv8r64.conf | 3 +- .../conf/machine/include/fvp-common.inc | 3 +- .../lib/oeqa/runtime/cases/fvp_devices.py | 130 ++++++++++++++++++ 3 files changed, 134 insertions(+), 2 deletions(-) create mode 100644 meta-arm/lib/oeqa/runtime/cases/fvp_devices.py diff --git a/meta-arm-bsp/conf/machine/fvp-baser-aemv8r64.conf b/meta-arm-bsp/conf/machine/fvp-baser-aemv8r64.conf index 7dbc53a1..c0ac3d75 100644 --- a/meta-arm-bsp/conf/machine/fvp-baser-aemv8r64.conf +++ b/meta-arm-bsp/conf/machine/fvp-baser-aemv8r64.conf @@ -30,9 +30,10 @@ MACHINE_EXTRA_RRECOMMENDS += "ssh-pregen-hostkeys" # testimage configuration TEST_TARGET = "OEFVPTarget" -TEST_SUITES:append = " fvp_boot" +TEST_SUITES:append = " fvp_boot fvp_devices" TEST_TARGET_IP ?= "127.0.0.1:8022" TEST_SERVER_IP ?= "127.0.1.1" +TEST_FVP_DEVICES ?= "rtc watchdog networking virtiorng cpu_hotplug" FVP_EXTRA_ARGS = "-a cluster0*=linux-system.axf" FVP_PROVIDER ?= "fvp-base-r-aem-native" diff --git a/meta-arm-bsp/conf/machine/include/fvp-common.inc b/meta-arm-bsp/conf/machine/include/fvp-common.inc index f80ac4c2..b76de3c4 100644 --- a/meta-arm-bsp/conf/machine/include/fvp-common.inc +++ b/meta-arm-bsp/conf/machine/include/fvp-common.inc @@ -24,7 +24,8 @@ MACHINE_EXTRA_RRECOMMENDS += "ssh-pregen-hostkeys" TEST_TARGET = "OEFVPTarget" TEST_TARGET_IP = "127.0.0.1:8022" -TEST_SUITES:append = " fvp_boot" +TEST_SUITES:append = " fvp_boot fvp_devices" +TEST_FVP_DEVICES ?= "rtc watchdog networking virtiorng cpu_hotplug" FVP_PROVIDER ?= "fvp-base-a-aem-native" FVP_EXE ?= "FVP_Base_RevC-2xAEMvA" diff --git a/meta-arm/lib/oeqa/runtime/cases/fvp_devices.py b/meta-arm/lib/oeqa/runtime/cases/fvp_devices.py new file mode 100644 index 00000000..0246e76a --- /dev/null +++ b/meta-arm/lib/oeqa/runtime/cases/fvp_devices.py @@ -0,0 +1,130 @@ +from oeqa.runtime.case import OERuntimeTestCase +from oeqa.core.decorator.data import skipIfNotInDataVar +from oeqa.core.decorator.depends import OETestDepends + + +class FvpDevicesTest(OERuntimeTestCase): + def run_cmd(self, cmd, check=True): + """ + A wrapper around self.target.run, which: + * Fails the test on command failure by default + * Allows the "run" behavior to be overridden in sub-classes + """ + (status, output) = self.target.run(cmd) + if status and check: + self.fail("Command '%s' returned non-zero exit " + "status %d:\n%s" % (cmd, status, output)) + + return (status, output) + + def check_devices(self, cls, min_count, search_drivers): + # Find all the devices of the specified class + cmd = f'find "/sys/class/{cls}" -type l -maxdepth 1' + _, output = self.run_cmd(cmd) + + devices = output.split() + self.assertGreaterEqual(len(devices), + min_count, + msg='Device count is lower than expected') + + # Assert that at least one of the devices uses at least one of the + # drivers + drivers = set() + for device in devices: + cmd = f'basename "$(readlink "{device}/device/driver")"' + _, output = self.run_cmd(cmd) + drivers.update(output.split()) + + self.assertTrue(drivers & set(search_drivers), + msg='No device uses either of the drivers: ' + + str(search_drivers)) + + def check_rng(self, hw_random, dev): + cmd = f'cat {hw_random} | grep {dev}' + self.run_cmd(cmd) + + def set_cpu(self, cpu_num, flag): + # Issue echo command + self.run_cmd( + f'echo "{flag}" > "/sys/devices/system/cpu/cpu{cpu_num}/online"', + check = False, + ) + _, output = self.run_cmd( + f'cat "/sys/devices/system/cpu/cpu{cpu_num}/online"' + ) + + return output == flag + + def enable_cpu(self, cpu_num): + return self.set_cpu(cpu_num, "1") + + def disable_cpu(self, cpu_num): + return self.set_cpu(cpu_num, "0") + + @OETestDepends(['ssh.SSHTest.test_ssh']) + @skipIfNotInDataVar('TEST_FVP_DEVICES', 'cpu_hotplug', + 'cpu_hotplug not included in BSP tests') + def test_cpu_hotplug(self): + _, cpus = self.run_cmd('find /sys/firmware/devicetree/base/cpus/' + ' -name "cpu@*" -maxdepth 1 | wc -l') + + try: + count_cpus = int(cpus) + except ValueError: + self.fail(f"Expected number of CPUs, but found this:\n{cpus}") + + self.num_cpus = int(self.td.get('TEST_CPU_HOTPLUG_NUM_CPUS', + count_cpus)) + try: + # Test that all cores are online + _, cpus = self.run_cmd('grep -c "processor" /proc/cpuinfo') + self.assertEqual(int(cpus), self.num_cpus) + # Don't try to disable here the only cpu present in the system. + if self.num_cpus > 1: + # Test that we can stop each core individually + for i in range(self.num_cpus): + self.assertTrue(self.disable_cpu(i)) + self.assertTrue(self.enable_cpu(i)) + + # Test that we cannot disable all cores + for i in range(self.num_cpus - 1): + self.assertTrue(self.disable_cpu(i)) + # Disabling last core should trigger an error + self.assertFalse(self.disable_cpu(self.num_cpus - 1)) + finally: + # Ensure all CPUs are re-enabled + for i in range(self.num_cpus): + self.enable_cpu(i) + + @OETestDepends(['ssh.SSHTest.test_ssh']) + @skipIfNotInDataVar('TEST_FVP_DEVICES', 'rtc', + 'rtc device not included in BSP tests') + def test_rtc(self): + self.check_devices("rtc", 1, ["rtc-pl031"]) + self.run_cmd('hwclock') + + @OETestDepends(['ssh.SSHTest.test_ssh']) + @skipIfNotInDataVar('TEST_FVP_DEVICES', 'watchdog', + 'watchdog device not included in BSP tests') + def test_watchdog(self): + self.check_devices("watchdog", 1, ["sp805-wdt", "sbsa-gwdt"]) + + @OETestDepends(['ssh.SSHTest.test_ssh']) + @skipIfNotInDataVar('TEST_FVP_DEVICES', 'networking', + 'networking device not included in BSP tests') + def test_networking(self): + self.check_devices("net", 2, ["virtio_net", "vif"]) + + # Check that outbound network connections work + self.run_cmd('wget -O /dev/null "https://www.arm.com"') + + @OETestDepends(['ssh.SSHTest.test_ssh']) + @skipIfNotInDataVar('TEST_FVP_DEVICES', 'virtiorng', + 'virtiorng device not included in BSP tests') + def test_virtiorng(self): + self.check_rng('/sys/devices/virtual/misc/hw_random/rng_available', + 'virtio_rng.0') + self.check_rng('/sys/devices/virtual/misc/hw_random/rng_current', + 'virtio_rng.0') + + self.run_cmd('hexdump -n 32 /dev/hwrng')