[6/6] arm/oeqa: increase optee test timeout

OPTEE tests are failing in CI on slower systems due to timing out, but
actually finish when given enough time to complete.  Increase the
timeout value to be roughly 100 seconds longer than the time it is
currently taking to finish on the slower systems.

Signed-off-by: Jon Mason <jon.mason@arm.com>
---
 meta-arm/lib/oeqa/runtime/cases/optee.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

On Tue, May 07, 2024 at 01:33:13PM -0400, Jon Mason wrote:
> OPTEE tests are failing in CI on slower systems due to timing out, but
> actually finish when given enough time to complete.  Increase the
> timeout value to be roughly 100 seconds longer than the time it is
> currently taking to finish on the slower systems.
> 
> Signed-off-by: Jon Mason <jon.mason@arm.com>

This should have:
Fixes: d4507866677c ("oeqa runtime: add optee.py test")

Since it is trivial, I'm just going to add to the commit manually
(instead of pushing a v2 for just this).

Thanks,
Jon

> ---
>  meta-arm/lib/oeqa/runtime/cases/optee.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/meta-arm/lib/oeqa/runtime/cases/optee.py b/meta-arm/lib/oeqa/runtime/cases/optee.py
> index 0a0bc31ed3d8..73bb3894e106 100644
> --- a/meta-arm/lib/oeqa/runtime/cases/optee.py
> +++ b/meta-arm/lib/oeqa/runtime/cases/optee.py
> @@ -12,7 +12,7 @@ class OpteeTestSuite(OERuntimeTestCase):
>      """
>      Run OP-TEE tests (xtest).
>      """
> -    @OETimeout(800)
> +    @OETimeout(1100)
>      @OEHasPackage(['optee-test'])
>      def test_opteetest_xtest(self):
>          # clear storage before executing tests
> @@ -20,5 +20,5 @@ class OpteeTestSuite(OERuntimeTestCase):
>          status, output = self.target.run(cmd, timeout=60)
>          self.assertEqual(status, 0, msg='\n'.join([cmd, output]))
>          cmd = "xtest"
> -        status, output = self.target.run(cmd, timeout=600)
> +        status, output = self.target.run(cmd, timeout=1000)
>          self.assertEqual(status, 0, msg='\n'.join([cmd, output]))
> -- 
> 2.39.3 (Apple Git-146)
> 
>

Hi,

On Tue, May 07, 2024 at 02:21:36PM -0400, Jon Mason via lists.yoctoproject.org wrote:
> On Tue, May 07, 2024 at 01:33:13PM -0400, Jon Mason wrote:
> > OPTEE tests are failing in CI on slower systems due to timing out, but
> > actually finish when given enough time to complete.  Increase the
> > timeout value to be roughly 100 seconds longer than the time it is
> > currently taking to finish on the slower systems.
> > 
> > Signed-off-by: Jon Mason <jon.mason@arm.com>
> 
> This should have:
> Fixes: d4507866677c ("oeqa runtime: add optee.py test")

Yes, the optee xtest execution time depends heavily on the target machine,
real or emulated with qemu. Also, the number of tests and thus execution time
are increasing in every optee release.

Acked-by: Mikko Rapeli <mikko.rapeli@linaro.org>

Cheers,

-Mikko

Hi,

On Wed, May 08, 2024 at 11:07:20AM +0300, Mikko Rapeli via lists.yoctoproject.org wrote:
> Hi,
> 
> On Tue, May 07, 2024 at 02:21:36PM -0400, Jon Mason via lists.yoctoproject.org wrote:
> > On Tue, May 07, 2024 at 01:33:13PM -0400, Jon Mason wrote:
> > > OPTEE tests are failing in CI on slower systems due to timing out, but
> > > actually finish when given enough time to complete.  Increase the
> > > timeout value to be roughly 100 seconds longer than the time it is
> > > currently taking to finish on the slower systems.
> > > 
> > > Signed-off-by: Jon Mason <jon.mason@arm.com>
> > 
> > This should have:
> > Fixes: d4507866677c ("oeqa runtime: add optee.py test")
> 
> Yes, the optee xtest execution time depends heavily on the target machine,
> real or emulated with qemu. Also, the number of tests and thus execution time
> are increasing in every optee release.
> 
> Acked-by: Mikko Rapeli <mikko.rapeli@linaro.org>

Maybe worth mentioning that with our Trusted Substrate firmware

https://gitlab.com/Linaro/trustedsubstrate/meta-ts/

which has TPM support and UEFI secure boot configured with tf-a, u-boot etc
for multiple boards, compared to qemuarm64-secureboot and qemuarm-secureboot
in meta-arm, we had some hangs with this test under oeqa and qemu.
Same hangs did not happen on real HW. Something to do with serial port
handling in qemu and how oeqa python framework uses them. Thus we set the timeouts
to be quite close to the real execution time to avoid long delays in detecting the
error situation. Good thing is that optee xtest does allow to skip/filter out tests
which are known to cause problems.

https://gitlab.com/Linaro/trusted-reference-stack/trs/-/blob/main/meta-trs/lib/oeqa/runtime/cases/opteetest.py?ref_type=heads

Cheers,

-Mikko

On Wed, May 08, 2024 at 11:32:39AM +0300, Mikko Rapeli wrote:
> Hi,
> 
> On Wed, May 08, 2024 at 11:07:20AM +0300, Mikko Rapeli via lists.yoctoproject.org wrote:
> > Hi,
> > 
> > On Tue, May 07, 2024 at 02:21:36PM -0400, Jon Mason via lists.yoctoproject.org wrote:
> > > On Tue, May 07, 2024 at 01:33:13PM -0400, Jon Mason wrote:
> > > > OPTEE tests are failing in CI on slower systems due to timing out, but
> > > > actually finish when given enough time to complete.  Increase the
> > > > timeout value to be roughly 100 seconds longer than the time it is
> > > > currently taking to finish on the slower systems.
> > > > 
> > > > Signed-off-by: Jon Mason <jon.mason@arm.com>
> > > 
> > > This should have:
> > > Fixes: d4507866677c ("oeqa runtime: add optee.py test")
> > 
> > Yes, the optee xtest execution time depends heavily on the target machine,
> > real or emulated with qemu. Also, the number of tests and thus execution time
> > are increasing in every optee release.
> > 
> > Acked-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> 
> Maybe worth mentioning that with our Trusted Substrate firmware
> 
> https://gitlab.com/Linaro/trustedsubstrate/meta-ts/
> 
> which has TPM support and UEFI secure boot configured with tf-a, u-boot etc
> for multiple boards, compared to qemuarm64-secureboot and qemuarm-secureboot
> in meta-arm, we had some hangs with this test under oeqa and qemu.
> Same hangs did not happen on real HW. Something to do with serial port
> handling in qemu and how oeqa python framework uses them. Thus we set the timeouts
> to be quite close to the real execution time to avoid long delays in detecting the
> error situation. Good thing is that optee xtest does allow to skip/filter out tests
> which are known to cause problems.
> 
> https://gitlab.com/Linaro/trusted-reference-stack/trs/-/blob/main/meta-trs/lib/oeqa/runtime/cases/opteetest.py?ref_type=heads
> 
> Cheers,
> 
> -Mikko
> 

It's still occasionally timing out in my personal CI, with ancient
systems.  I'm thinking about holding off on this patch and seeing if I
can't find a better length of time (or get rid of my ancient systems).

Thanks for the confirmation that it's being seen on your end too.

Thanks,
Jon