diff mbox series

[2/2] oeqa/postactions: do not uncompress retrieved archive on host

Message ID 20240812151404.134030-3-alexis.lothore@bootlin.com
State Accepted, archived
Commit f90894d996c8a8f980e46c87b7968b176793b3fe
Headers show
Series oeqa/postactions: save ptests artifacts as archive on host | expand

Commit Message

Alexis Lothoré Aug. 12, 2024, 3:14 p.m. UTC 
From: Alexis Lothoré <alexis.lothore@bootlin.com>

Current postaction module executes a remote tar command, pipe it in a SSH
connection, and uncompress the raw stream with another tar command. With
this command, the whole artifacts tree is directly available on the host
executing the test, but it is not very convenient if we want to download
the whole retrieved ptests directory.

Stop uncompressing the retrieved ptests archive onto host, just save the
archive as it is. The new output then looks like the following:

tmp/log/oeqa-artefacts
└── oeqa-target-artefacts-20240812-juzqdb80
    ├── host_disk_usage.txt
    ├── target_disk_usage.txt
    └── tests_artifacts.tar.gz

Suggested-By: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
---
 meta/lib/oeqa/utils/postactions.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Comments

Richard Purdie Sept. 19, 2024, 7:42 a.m. UTC | #1 
On Mon, 2024-08-12 at 17:14 +0200, Alexis Lothoré via lists.openembedded.org wrote:
> From: Alexis Lothoré <alexis.lothore@bootlin.com>
> 
> Current postaction module executes a remote tar command, pipe it in a SSH
> connection, and uncompress the raw stream with another tar command. With
> this command, the whole artifacts tree is directly available on the host
> executing the test, but it is not very convenient if we want to download
> the whole retrieved ptests directory.
> 
> Stop uncompressing the retrieved ptests archive onto host, just save the
> archive as it is. The new output then looks like the following:
> 
> tmp/log/oeqa-artefacts
> └── oeqa-target-artefacts-20240812-juzqdb80
>     ├── host_disk_usage.txt
>     ├── target_disk_usage.txt
>     └── tests_artifacts.tar.gz
> 
> Suggested-By: Alexandre Belloni <alexandre.belloni@bootlin.com>
> Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
> ---
>  meta/lib/oeqa/utils/postactions.py | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/lib/oeqa/utils/postactions.py b/meta/lib/oeqa/utils/postactions.py
> index 3b537146c50a..d5080523aa83 100644
> --- a/meta/lib/oeqa/utils/postactions.py
> +++ b/meta/lib/oeqa/utils/postactions.py
> @@ -68,7 +68,8 @@ def list_and_fetch_failed_tests_artifacts(d, tc, artifacts_list, outputdir):
>          (status, output) = tc.target.run(cmd, raw = True)
>          if status != 0 or not output:
>              raise Exception("Error while fetching compressed artifacts")
> -        p = subprocess.run(["tar", "zxf", "-", "-C", outputdir], input=output)
> +        with open(archive_name, "wb") as f:
> +            f.write(output)
>      except Exception as e:
>          bb.warn(f"Can not retrieve artifacts from test target: {e}")


This change is causing failures since archive_name isn't defined anywhere :/

Cheers,

Richard
Richard Purdie Sept. 19, 2024, 12:58 p.m. UTC | #2 
On Thu, 2024-09-19 at 08:42 +0100, Richard Purdie via
lists.openembedded.org wrote:
> On Mon, 2024-08-12 at 17:14 +0200, Alexis Lothoré via
> lists.openembedded.org wrote:
> > From: Alexis Lothoré <alexis.lothore@bootlin.com>
> > 
> > Current postaction module executes a remote tar command, pipe it in
> > a SSH
> > connection, and uncompress the raw stream with another tar command.
> > With
> > this command, the whole artifacts tree is directly available on the
> > host
> > executing the test, but it is not very convenient if we want to
> > download
> > the whole retrieved ptests directory.
> > 
> > Stop uncompressing the retrieved ptests archive onto host, just
> > save the
> > archive as it is. The new output then looks like the following:
> > 
> > tmp/log/oeqa-artefacts
> > └── oeqa-target-artefacts-20240812-juzqdb80
> >     ├── host_disk_usage.txt
> >     ├── target_disk_usage.txt
> >     └── tests_artifacts.tar.gz
> > 
> > Suggested-By: Alexandre Belloni <alexandre.belloni@bootlin.com>
> > Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
> > ---
> >  meta/lib/oeqa/utils/postactions.py | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > diff --git a/meta/lib/oeqa/utils/postactions.py
> > b/meta/lib/oeqa/utils/postactions.py
> > index 3b537146c50a..d5080523aa83 100644
> > --- a/meta/lib/oeqa/utils/postactions.py
> > +++ b/meta/lib/oeqa/utils/postactions.py
> > @@ -68,7 +68,8 @@ def list_and_fetch_failed_tests_artifacts(d, tc,
> > artifacts_list, outputdir):
> >          (status, output) = tc.target.run(cmd, raw = True)
> >          if status != 0 or not output:
> >              raise Exception("Error while fetching compressed
> > artifacts")
> > -        p = subprocess.run(["tar", "zxf", "-", "-C", outputdir],
> > input=output)
> > +        with open(archive_name, "wb") as f:
> > +            f.write(output)
> >      except Exception as e:
> >          bb.warn(f"Can not retrieve artifacts from test target:
> > {e}")
> 
> 
> This change is causing failures since archive_name isn't defined
> anywhere :/

I'd also note that scarthgap actually throws errors:

https://valkyrie.yoctoproject.org/#/builders/73/builds/135/steps/12/logs/stdio

Exception: NameError: name 'artifact_path' is not defined

which is related to some of these patches :/.

Cheers,

Richard
Alexis Lothoré Sept. 19, 2024, 2:29 p.m. UTC | #3 
Hello Richard,

On 9/19/24 14:58, Richard Purdie wrote:
> On Thu, 2024-09-19 at 08:42 +0100, Richard Purdie via
> lists.openembedded.org wrote:
>> On Mon, 2024-08-12 at 17:14 +0200, Alexis Lothoré via
>> lists.openembedded.org wrote:
>>> From: Alexis Lothoré <alexis.lothore@bootlin.com>

[...]

>>> --- a/meta/lib/oeqa/utils/postactions.py
>>> +++ b/meta/lib/oeqa/utils/postactions.py
>>> @@ -68,7 +68,8 @@ def list_and_fetch_failed_tests_artifacts(d, tc,
>>> artifacts_list, outputdir):
>>>          (status, output) = tc.target.run(cmd, raw = True)
>>>          if status != 0 or not output:
>>>              raise Exception("Error while fetching compressed
>>> artifacts")
>>> -        p = subprocess.run(["tar", "zxf", "-", "-C", outputdir],
>>> input=output)
>>> +        with open(archive_name, "wb") as f:
>>> +            f.write(output)
>>>      except Exception as e:
>>>          bb.warn(f"Can not retrieve artifacts from test target:
>>> {e}")
>>
>>
>> This change is causing failures since archive_name isn't defined
>> anywhere :/
> 
> I'd also note that scarthgap actually throws errors:
> 
> https://valkyrie.yoctoproject.org/#/builders/73/builds/135/steps/12/logs/stdio
> 
> Exception: NameError: name 'artifact_path' is not defined
> 
> which is related to some of these patches :/.

I am out of  office right now with limited access to computer (I come back next
week but then I will be traveling to attend Kernel Recipes). If no one fixes it
before, I'll take a look and send a fix once I return to the office (end of next
week)

Alexis
Alexandre Belloni Sept. 19, 2024, 3 p.m. UTC | #4 
On 19/09/2024 16:29:06+0200, Alexis Lothor� wrote:
> Hello Richard,
> 
> On 9/19/24 14:58, Richard Purdie wrote:
> > On Thu, 2024-09-19 at 08:42 +0100, Richard Purdie via
> > lists.openembedded.org wrote:
> >> On Mon, 2024-08-12 at 17:14 +0200, Alexis Lothor� via
> >> lists.openembedded.org wrote:
> >>> From: Alexis Lothor� <alexis.lothore@bootlin.com>
> 
> [...]
> 
> >>> --- a/meta/lib/oeqa/utils/postactions.py
> >>> +++ b/meta/lib/oeqa/utils/postactions.py
> >>> @@ -68,7 +68,8 @@ def list_and_fetch_failed_tests_artifacts(d, tc,
> >>> artifacts_list, outputdir):
> >>> �������� (status, output) = tc.target.run(cmd, raw = True)
> >>> �������� if status != 0 or not output:
> >>> ������������ raise Exception("Error while fetching compressed
> >>> artifacts")
> >>> -������� p = subprocess.run(["tar", "zxf", "-", "-C", outputdir],
> >>> input=output)
> >>> +������� with open(archive_name, "wb") as f:
> >>> +����������� f.write(output)
> >>> ���� except Exception as e:
> >>> �������� bb.warn(f"Can not retrieve artifacts from test target:
> >>> {e}")
> >>
> >>
> >> This change is causing failures since archive_name isn't defined
> >> anywhere :/
> > 
> > I'd also note that scarthgap actually throws errors:
> > 
> > https://valkyrie.yoctoproject.org/#/builders/73/builds/135/steps/12/logs/stdio
> > 
> > Exception: NameError: name 'artifact_path' is not defined
> > 
> > which is related to some of these patches :/.
> 
> I am out of  office right now with limited access to computer (I come back next
> week but then I will be traveling to attend Kernel Recipes). If no one fixes it
> before, I'll take a look and send a fix once I return to the office (end of next
> week)
> 

Mathieu is going to take care of this tomorrow/beginning of next week.
diff mbox series

Patch

diff --git a/meta/lib/oeqa/utils/postactions.py b/meta/lib/oeqa/utils/postactions.py
index 3b537146c50a..d5080523aa83 100644
--- a/meta/lib/oeqa/utils/postactions.py
+++ b/meta/lib/oeqa/utils/postactions.py
@@ -68,7 +68,8 @@  def list_and_fetch_failed_tests_artifacts(d, tc, artifacts_list, outputdir):
         (status, output) = tc.target.run(cmd, raw = True)
         if status != 0 or not output:
             raise Exception("Error while fetching compressed artifacts")
-        p = subprocess.run(["tar", "zxf", "-", "-C", outputdir], input=output)
+        with open(archive_name, "wb") as f:
+            f.write(output)
     except Exception as e:
         bb.warn(f"Can not retrieve artifacts from test target: {e}")