| Message ID | 20230113150646.44144-1-alexis.lothore@bootlin.com |
|---|---|
| Headers | show |
| Series | generate regression reports against proper releases | expand |
On Fri, 2023-01-13 at 16:06 +0100, Alexis Lothoré via lists.yoctoproject.org wrote: > This work has been initiated while discussing issue #14065 > (https://bugzilla.yoctoproject.org/show_bug.cgi?id=14065) > The topic initially covers ptest reports between releases, but after discussions > with R. Purdie (https://lists.yoctoproject.org/g/automated-testing/message/1211), > a general regression report reliability topic emerged. Those first > patches aims to make regression reports more useful during releases by computing > against which revision it would be relevant to check for regressions. > > A few notes about those patches : > - Since it affects release process, I struggled to properly test the new behavior, > so testing has been made by stubbing send-qa-email side effects and picking some > samples of build properties and layerinfo.json from autobuilder history to run > manually send-qa-email on my machine. But I will gladly take into account any > advice or procedure to test it more thoroughly > - I also have some uncommitted python unit tests on "previous version > computation", but current scripts directory does not allow proper unit tests > integration. If appropriate, I could submit some minor reworks to make it viable > for testing and submit those tests I'm glad you mentioned this! Yes, I think it is important we add some tests for this. We can rework these scripts as needed as the project is their main user at this point. > - This new regression checking introduced the need to clone a wider testresults > history, which is (very) slow on my machine, but I do not now how slow it is > on autobuilder. Could it be an issue for the delivery process ? Does it need to > be smarter on this point ? I think it will be slow on the autobuilder and we're going to have to do something different to make this work well. My clone still hasn't finished and is at 5.75GB of data now (15 mins to get the last 4.5GB) and still going. > - If appropriate, new patch could follow to allow regression reports generation > against multiple revisions : for example we could check for a milestone build > against previous milestone AND previous major release I think that is the direction we need to go so yes please! Cheers, Richard
Hi Richard, On 13/01/2023 19:01, Richard Purdie wrote: > On Fri, 2023-01-13 at 16:06 +0100, Alexis Lothoré via > lists.yoctoproject.org wrote: >> - I also have some uncommitted python unit tests on "previous version >> computation", but current scripts directory does not allow proper >> unit tests >> integration. If appropriate, I could submit some minor reworks to make >> it viable >> for testing and submit those tests > > I'm glad you mentioned this! Yes, I think it is important we add some > tests for this. We can rework these scripts as needed as the project is > their main user at this point. Good, I will then add some minor rework to allow introducing unit tests alongside the version computation in the next revision > >> - This new regression checking introduced the need to clone a wider >> testresults >> history, which is (very) slow on my machine, but I do not now how >> slow it is >> on autobuilder. Could it be an issue for the delivery process ? Does >> it need to >> be smarter on this point ? > > I think it will be slow on the autobuilder and we're going to have to > do something different to make this work well. My clone still hasn't > finished and is at 5.75GB of data now (15 mins to get the last 4.5GB) > and still going. OK. I observed a quite similar behaviour while pulling manually the whole repository, but I thought it might be linked to network issue on my side. I will add a smarter way based on your suggestion (git ls-remote) >> - If appropriate, new patch could follow to allow regression reports >> generation >> against multiple revisions : for example we could check for a >> milestone build >> against previous milestone AND previous major release > > I think that is the direction we need to go so yes please! Noted. If it is OK for you, I will keep this last point for another set of patches, once this initial work is approved and merged. Regards,
This work has been initiated while discussing issue #14065 (https://bugzilla.yoctoproject.org/show_bug.cgi?id=14065) The topic initially covers ptest reports between releases, but after discussions with R. Purdie (https://lists.yoctoproject.org/g/automated-testing/message/1211), a general regression report reliability topic emerged. Those first patches aims to make regression reports more useful during releases by computing against which revision it would be relevant to check for regressions. A few notes about those patches : - Since it affects release process, I struggled to properly test the new behavior, so testing has been made by stubbing send-qa-email side effects and picking some samples of build properties and layerinfo.json from autobuilder history to run manually send-qa-email on my machine. But I will gladly take into account any advice or procedure to test it more thoroughly - I also have some uncommitted python unit tests on "previous version computation", but current scripts directory does not allow proper unit tests integration. If appropriate, I could submit some minor reworks to make it viable for testing and submit those tests - This new regression checking introduced the need to clone a wider testresults history, which is (very) slow on my machine, but I do not now how slow it is on autobuilder. Could it be an issue for the delivery process ? Does it need to be smarter on this point ? - If appropriate, new patch could follow to allow regression reports generation against multiple revisions : for example we could check for a milestone build against previous milestone AND previous major release Alexis Lothoré (2): scripts/send-qa-email : clone older history to allow comparing against older releases scripts/send-qa-email: Generate regression reports against most relevant release scripts/send-qa-email | 69 ++++++++++++++++++++++++++++++++----------- scripts/utils.py | 47 +++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+), 17 deletions(-)