| Message ID | 20221124025528.114101-1-Qi.Chen@windriver.com |
|---|---|
| State | Accepted, archived |
| Commit | df9a2b69748d8a24c3390f812225231e9e9acb66 |
| Headers | show |
| Series | selftest: allow '-R' and '-r' be used together | expand |
Hello, On 23/11/2022 18:55:28-0800, Chen Qi wrote: > For now, there's no easy way to skip some specified tests when > using 'oe-selftest -r'. This is because '-R' and '-r' are in the > mutually exclusive group. Looking at the codes, the codes have already > allowed running specified tests and skipping specified tests at > the same time. So move '-R' out of the mutually exclusive group > and change its help message. > > After this change, the following command could be used. > oe-selftest -r A -R A.B.C > > This does introduce a behavior regression. > Previous 'oe-selftest -R xxx' needs to now be changed to > 'oe-selftest -a -R xxx'. > This fails on the autobuilders which uses: oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15 https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4395/steps/14/logs/stdio > Signed-off-by: Chen Qi <Qi.Chen@windriver.com> > --- > meta/lib/oeqa/selftest/context.py | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > > diff --git a/meta/lib/oeqa/selftest/context.py b/meta/lib/oeqa/selftest/context.py > index 78c7a467e2..c7dd03ce37 100644 > --- a/meta/lib/oeqa/selftest/context.py > +++ b/meta/lib/oeqa/selftest/context.py > @@ -154,9 +154,6 @@ class OESelftestTestContextExecutor(OETestContextExecutor): > group.add_argument('-a', '--run-all-tests', default=False, > action="store_true", dest="run_all_tests", > help='Run all (unhidden) tests') > - group.add_argument('-R', '--skip-tests', required=False, action='store', > - nargs='+', dest="skips", default=None, > - help='Run all (unhidden) tests except the ones specified. Format should be <module>[.<class>[.<test_method>]]') > group.add_argument('-r', '--run-tests', required=False, action='store', > nargs='+', dest="run_tests", default=None, > help='Select what tests to run (modules, classes or test methods). Format should be: <module>.<class>.<test_method>') > @@ -171,6 +168,9 @@ class OESelftestTestContextExecutor(OETestContextExecutor): > action="store_true", default=False, > help='List all available tests.') > > + parser.add_argument('-R', '--skip-tests', required=False, action='store', > + nargs='+', dest="skips", default=None, > + help='Skip the tests specified. Format should be <module>[.<class>[.<test_method>]]') > parser.add_argument('-j', '--num-processes', dest='processes', action='store', > type=int, help="number of processes to execute in parallel with") > > -- > 2.37.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#173732): https://lists.openembedded.org/g/openembedded-core/message/173732 > Mute This Topic: https://lists.openembedded.org/mt/95231396/3617179 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Hi Alexandre, Yes, I'm aware of this behavior regression. With this patch, 'oe-selftest -R xxx' needs to be changed to 'oe-selftest -a -R xxx'. I'm not sure if this is useful for other people. Because how people use oe-selftest is unknown to me. For me, I want to run something like 'oe-selftest -r A -R A.B.C'. Regards, Qi -----Original Message----- From: Alexandre Belloni <alexandre.belloni@bootlin.com> Sent: Thursday, November 24, 2022 7:35 PM To: Chen, Qi <Qi.Chen@windriver.com> Cc: openembedded-core@lists.openembedded.org Subject: Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together Hello, On 23/11/2022 18:55:28-0800, Chen Qi wrote: > For now, there's no easy way to skip some specified tests when using > 'oe-selftest -r'. This is because '-R' and '-r' are in the mutually > exclusive group. Looking at the codes, the codes have already allowed > running specified tests and skipping specified tests at the same time. > So move '-R' out of the mutually exclusive group and change its help > message. > > After this change, the following command could be used. > oe-selftest -r A -R A.B.C > > This does introduce a behavior regression. > Previous 'oe-selftest -R xxx' needs to now be changed to 'oe-selftest > -a -R xxx'. > This fails on the autobuilders which uses: oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15 https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4395/steps/14/logs/stdio > Signed-off-by: Chen Qi <Qi.Chen@windriver.com> > --- > meta/lib/oeqa/selftest/context.py | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > > diff --git a/meta/lib/oeqa/selftest/context.py > b/meta/lib/oeqa/selftest/context.py > index 78c7a467e2..c7dd03ce37 100644 > --- a/meta/lib/oeqa/selftest/context.py > +++ b/meta/lib/oeqa/selftest/context.py > @@ -154,9 +154,6 @@ class OESelftestTestContextExecutor(OETestContextExecutor): > group.add_argument('-a', '--run-all-tests', default=False, > action="store_true", dest="run_all_tests", > help='Run all (unhidden) tests') > - group.add_argument('-R', '--skip-tests', required=False, action='store', > - nargs='+', dest="skips", default=None, > - help='Run all (unhidden) tests except the ones specified. Format should be <module>[.<class>[.<test_method>]]') > group.add_argument('-r', '--run-tests', required=False, action='store', > nargs='+', dest="run_tests", default=None, > help='Select what tests to run (modules, classes or > test methods). Format should be: <module>.<class>.<test_method>') @@ -171,6 +168,9 @@ class OESelftestTestContextExecutor(OETestContextExecutor): > action="store_true", default=False, > help='List all available tests.') > > + parser.add_argument('-R', '--skip-tests', required=False, action='store', > + nargs='+', dest="skips", default=None, > + help='Skip the tests specified. Format should be > + <module>[.<class>[.<test_method>]]') > parser.add_argument('-j', '--num-processes', dest='processes', action='store', > type=int, help="number of processes to execute in > parallel with") > > -- > 2.37.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#173732): https://lists.openembedded.org/g/openembedded-core/message/173732 > Mute This Topic: https://lists.openembedded.org/mt/95231396/3617179 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On 24/11/2022 12:24:38+0000, Chen Qi wrote: > Hi Alexandre, > > Yes, I'm aware of this behavior regression. > With this patch, 'oe-selftest -R xxx' needs to be changed to 'oe-selftest -a -R xxx'. > Yes, my point is that there is no -R in the command line used by the AB > I'm not sure if this is useful for other people. Because how people use oe-selftest is unknown to me. > For me, I want to run something like 'oe-selftest -r A -R A.B.C'. > > Regards, > Qi > > -----Original Message----- > From: Alexandre Belloni <alexandre.belloni@bootlin.com> > Sent: Thursday, November 24, 2022 7:35 PM > To: Chen, Qi <Qi.Chen@windriver.com> > Cc: openembedded-core@lists.openembedded.org > Subject: Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together > > Hello, > > On 23/11/2022 18:55:28-0800, Chen Qi wrote: > > For now, there's no easy way to skip some specified tests when using > > 'oe-selftest -r'. This is because '-R' and '-r' are in the mutually > > exclusive group. Looking at the codes, the codes have already allowed > > running specified tests and skipping specified tests at the same time. > > So move '-R' out of the mutually exclusive group and change its help > > message. > > > > After this change, the following command could be used. > > oe-selftest -r A -R A.B.C > > > > This does introduce a behavior regression. > > Previous 'oe-selftest -R xxx' needs to now be changed to 'oe-selftest > > -a -R xxx'. > > > > This fails on the autobuilders which uses: > > oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15 > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4395/steps/14/logs/stdio > > > Signed-off-by: Chen Qi <Qi.Chen@windriver.com> > > --- > > meta/lib/oeqa/selftest/context.py | 6 +++--- > > 1 file changed, 3 insertions(+), 3 deletions(-) > > > > diff --git a/meta/lib/oeqa/selftest/context.py > > b/meta/lib/oeqa/selftest/context.py > > index 78c7a467e2..c7dd03ce37 100644 > > --- a/meta/lib/oeqa/selftest/context.py > > +++ b/meta/lib/oeqa/selftest/context.py > > @@ -154,9 +154,6 @@ class OESelftestTestContextExecutor(OETestContextExecutor): > > group.add_argument('-a', '--run-all-tests', default=False, > > action="store_true", dest="run_all_tests", > > help='Run all (unhidden) tests') > > - group.add_argument('-R', '--skip-tests', required=False, action='store', > > - nargs='+', dest="skips", default=None, > > - help='Run all (unhidden) tests except the ones specified. Format should be <module>[.<class>[.<test_method>]]') > > group.add_argument('-r', '--run-tests', required=False, action='store', > > nargs='+', dest="run_tests", default=None, > > help='Select what tests to run (modules, classes or > > test methods). Format should be: <module>.<class>.<test_method>') @@ -171,6 +168,9 @@ class OESelftestTestContextExecutor(OETestContextExecutor): > > action="store_true", default=False, > > help='List all available tests.') > > > > + parser.add_argument('-R', '--skip-tests', required=False, action='store', > > + nargs='+', dest="skips", default=None, > > + help='Skip the tests specified. Format should be > > + <module>[.<class>[.<test_method>]]') > > parser.add_argument('-j', '--num-processes', dest='processes', action='store', > > type=int, help="number of processes to execute in > > parallel with") > > > > -- > > 2.37.1 > > > > > > > > > > > > -- > Alexandre Belloni, co-owner and COO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#173743): https://lists.openembedded.org/g/openembedded-core/message/173743 > Mute This Topic: https://lists.openembedded.org/mt/95231396/3617179 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Hi Alexandre, The command line is: oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15 The '--skip-tests' option is the long format of '-R'. Regards, Qi -----Original Message----- From: Alexandre Belloni <alexandre.belloni@bootlin.com> Sent: Thursday, November 24, 2022 9:02 PM To: Chen, Qi <Qi.Chen@windriver.com> Cc: openembedded-core@lists.openembedded.org Subject: Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together On 24/11/2022 12:24:38+0000, Chen Qi wrote: > Hi Alexandre, > > Yes, I'm aware of this behavior regression. > With this patch, 'oe-selftest -R xxx' needs to be changed to 'oe-selftest -a -R xxx'. > Yes, my point is that there is no -R in the command line used by the AB > I'm not sure if this is useful for other people. Because how people use oe-selftest is unknown to me. > For me, I want to run something like 'oe-selftest -r A -R A.B.C'. > > Regards, > Qi > > -----Original Message----- > From: Alexandre Belloni <alexandre.belloni@bootlin.com> > Sent: Thursday, November 24, 2022 7:35 PM > To: Chen, Qi <Qi.Chen@windriver.com> > Cc: openembedded-core@lists.openembedded.org > Subject: Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used > together > > Hello, > > On 23/11/2022 18:55:28-0800, Chen Qi wrote: > > For now, there's no easy way to skip some specified tests when using > > 'oe-selftest -r'. This is because '-R' and '-r' are in the mutually > > exclusive group. Looking at the codes, the codes have already > > allowed running specified tests and skipping specified tests at the same time. > > So move '-R' out of the mutually exclusive group and change its help > > message. > > > > After this change, the following command could be used. > > oe-selftest -r A -R A.B.C > > > > This does introduce a behavior regression. > > Previous 'oe-selftest -R xxx' needs to now be changed to > > 'oe-selftest -a -R xxx'. > > > > This fails on the autobuilders which uses: > > oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg > buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T > machine -T toolchain-user -T toolchain-system -j 15 > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4395 > /steps/14/logs/stdio > > > Signed-off-by: Chen Qi <Qi.Chen@windriver.com> > > --- > > meta/lib/oeqa/selftest/context.py | 6 +++--- > > 1 file changed, 3 insertions(+), 3 deletions(-) > > > > diff --git a/meta/lib/oeqa/selftest/context.py > > b/meta/lib/oeqa/selftest/context.py > > index 78c7a467e2..c7dd03ce37 100644 > > --- a/meta/lib/oeqa/selftest/context.py > > +++ b/meta/lib/oeqa/selftest/context.py > > @@ -154,9 +154,6 @@ class OESelftestTestContextExecutor(OETestContextExecutor): > > group.add_argument('-a', '--run-all-tests', default=False, > > action="store_true", dest="run_all_tests", > > help='Run all (unhidden) tests') > > - group.add_argument('-R', '--skip-tests', required=False, action='store', > > - nargs='+', dest="skips", default=None, > > - help='Run all (unhidden) tests except the ones specified. Format should be <module>[.<class>[.<test_method>]]') > > group.add_argument('-r', '--run-tests', required=False, action='store', > > nargs='+', dest="run_tests", default=None, > > help='Select what tests to run (modules, classes or > > test methods). Format should be: <module>.<class>.<test_method>') @@ -171,6 +168,9 @@ class OESelftestTestContextExecutor(OETestContextExecutor): > > action="store_true", default=False, > > help='List all available tests.') > > > > + parser.add_argument('-R', '--skip-tests', required=False, action='store', > > + nargs='+', dest="skips", default=None, > > + help='Skip the tests specified. Format should be > > + <module>[.<class>[.<test_method>]]') > > parser.add_argument('-j', '--num-processes', dest='processes', action='store', > > type=int, help="number of processes to execute in > > parallel with") > > > > -- > > 2.37.1 > > > > > > > > > > > > -- > Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel > engineering https://bootlin.com > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#173743): > https://lists.openembedded.org/g/openembedded-core/message/173743 > Mute This Topic: https://lists.openembedded.org/mt/95231396/3617179 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub > [alexandre.belloni@bootlin.com] > -=-=-=-=-=-=-=-=-=-=-=- > -- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
On 24/11/2022 13:10:51+0000, Chen Qi wrote: > Hi Alexandre, > > The command line is: oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15 > > The '--skip-tests' option is the long format of '-R'. > Right, sorry about that :) I actually checked we didn't have any -R before sending the patch to the AB... I'll discuss that with RP. > Regards, > Qi > > -----Original Message----- > From: Alexandre Belloni <alexandre.belloni@bootlin.com> > Sent: Thursday, November 24, 2022 9:02 PM > To: Chen, Qi <Qi.Chen@windriver.com> > Cc: openembedded-core@lists.openembedded.org > Subject: Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together > > On 24/11/2022 12:24:38+0000, Chen Qi wrote: > > Hi Alexandre, > > > > Yes, I'm aware of this behavior regression. > > With this patch, 'oe-selftest -R xxx' needs to be changed to 'oe-selftest -a -R xxx'. > > > > Yes, my point is that there is no -R in the command line used by the AB > > > I'm not sure if this is useful for other people. Because how people use oe-selftest is unknown to me. > > For me, I want to run something like 'oe-selftest -r A -R A.B.C'. > > > > Regards, > > Qi > > > > -----Original Message----- > > From: Alexandre Belloni <alexandre.belloni@bootlin.com> > > Sent: Thursday, November 24, 2022 7:35 PM > > To: Chen, Qi <Qi.Chen@windriver.com> > > Cc: openembedded-core@lists.openembedded.org > > Subject: Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used > > together > > > > Hello, > > > > On 23/11/2022 18:55:28-0800, Chen Qi wrote: > > > For now, there's no easy way to skip some specified tests when using > > > 'oe-selftest -r'. This is because '-R' and '-r' are in the mutually > > > exclusive group. Looking at the codes, the codes have already > > > allowed running specified tests and skipping specified tests at the same time. > > > So move '-R' out of the mutually exclusive group and change its help > > > message. > > > > > > After this change, the following command could be used. > > > oe-selftest -r A -R A.B.C > > > > > > This does introduce a behavior regression. > > > Previous 'oe-selftest -R xxx' needs to now be changed to > > > 'oe-selftest -a -R xxx'. > > > > > > > This fails on the autobuilders which uses: > > > > oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg > > buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T > > machine -T toolchain-user -T toolchain-system -j 15 > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4395 > > /steps/14/logs/stdio > > > > > Signed-off-by: Chen Qi <Qi.Chen@windriver.com> > > > --- > > > meta/lib/oeqa/selftest/context.py | 6 +++--- > > > 1 file changed, 3 insertions(+), 3 deletions(-) > > > > > > diff --git a/meta/lib/oeqa/selftest/context.py > > > b/meta/lib/oeqa/selftest/context.py > > > index 78c7a467e2..c7dd03ce37 100644 > > > --- a/meta/lib/oeqa/selftest/context.py > > > +++ b/meta/lib/oeqa/selftest/context.py > > > @@ -154,9 +154,6 @@ class OESelftestTestContextExecutor(OETestContextExecutor): > > > group.add_argument('-a', '--run-all-tests', default=False, > > > action="store_true", dest="run_all_tests", > > > help='Run all (unhidden) tests') > > > - group.add_argument('-R', '--skip-tests', required=False, action='store', > > > - nargs='+', dest="skips", default=None, > > > - help='Run all (unhidden) tests except the ones specified. Format should be <module>[.<class>[.<test_method>]]') > > > group.add_argument('-r', '--run-tests', required=False, action='store', > > > nargs='+', dest="run_tests", default=None, > > > help='Select what tests to run (modules, classes or > > > test methods). Format should be: <module>.<class>.<test_method>') @@ -171,6 +168,9 @@ class OESelftestTestContextExecutor(OETestContextExecutor): > > > action="store_true", default=False, > > > help='List all available tests.') > > > > > > + parser.add_argument('-R', '--skip-tests', required=False, action='store', > > > + nargs='+', dest="skips", default=None, > > > + help='Skip the tests specified. Format should be > > > + <module>[.<class>[.<test_method>]]') > > > parser.add_argument('-j', '--num-processes', dest='processes', action='store', > > > type=int, help="number of processes to execute in > > > parallel with") > > > > > > -- > > > 2.37.1 > > > > > > > > > > > > > > > > > > > > -- > > Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel > > engineering https://bootlin.com > > > > > > > > > > -- > Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#173748): https://lists.openembedded.org/g/openembedded-core/message/173748 > Mute This Topic: https://lists.openembedded.org/mt/95231396/3617179 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Thu, 2022-11-24 at 14:35 +0100, Alexandre Belloni via lists.openembedded.org wrote: > On 24/11/2022 13:10:51+0000, Chen Qi wrote: > > Hi Alexandre, > > > > The command line is: oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15 > > > > The '--skip-tests' option is the long format of '-R'. > > > > Right, sorry about that :) I actually checked we didn't have any -R > before sending the patch to the AB... I'll discuss that with RP. The autobuilder is not running "all" tests though, it is running two groups of tests, -T machine and -T toolchain-user. Adding -a here doesn't really make sense. Cheers, Richard
Hi Richard, The AB is actually running 'all' tests. The '--skip-tests (-R)' option means 'Run all (unhidden) tests except the ones specified.', according to its help message. This is also its actual effect for now. You can see there's an implicit 'run all tests' meaning in this option. The '-T (--exclude-tag)' options means 'Exclude all (unhidden) tests that match any of the specified tag(s)', according to its help message. So the AB's oe-selftest command means: execute all tests except the ones tagged with 'machine' and 'toolchain-user', and also skip 'distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible'. Regards, Qi -----Original Message----- From: Richard Purdie <richard.purdie@linuxfoundation.org> Sent: Thursday, November 24, 2022 9:47 PM To: alexandre.belloni@bootlin.com; Chen, Qi <Qi.Chen@windriver.com> Cc: openembedded-core@lists.openembedded.org Subject: Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together On Thu, 2022-11-24 at 14:35 +0100, Alexandre Belloni via lists.openembedded.org wrote: > On 24/11/2022 13:10:51+0000, Chen Qi wrote: > > Hi Alexandre, > > > > The command line is: oe-selftest --skip-tests > > distrodata.Distrodata.test_checkpkg > > buildoptions.SourceMirroring.test_yocto_source_mirror reproducible > > -T machine -T toolchain-user -T toolchain-system -j 15 > > > > The '--skip-tests' option is the long format of '-R'. > > > > Right, sorry about that :) I actually checked we didn't have any -R > before sending the patch to the AB... I'll discuss that with RP. The autobuilder is not running "all" tests though, it is running two groups of tests, -T machine and -T toolchain-user. Adding -a here doesn't really make sense. Cheers, Richard
Hi Qi, On Fri, 2022-11-25 at 05:56 +0000, Chen Qi wrote: > The AB is actually running 'all' tests. > > The '--skip-tests (-R)' option means 'Run all (unhidden) tests except > the ones specified.', according to its help message. This is also its > actual effect for now. You can see there's an implicit 'run all > tests' meaning in this option. > The '-T (--exclude-tag)' options means 'Exclude all (unhidden) tests > that match any of the specified tag(s)', according to its help > message. > > So the AB's oe-selftest command means: execute all tests except the > ones tagged with 'machine' and 'toolchain-user', and also skip > 'distrodata.Distrodata.test_checkpkg > buildoptions.SourceMirroring.test_yocto_source_mirror reproducible'. You're right, a different build has -t machine and I'm confusing the two, sorry. I don't really like making changes which require lockstep changes to the autobuilder configuration but I can see why we could do with doing so here. I was thinking there should be symettry between -r and -R like there is with -T and -t but that also probably doesn't make sense when you think about it more. Alex: We'll probably have to work out how to make this change... Cheers, Richard
On 2022-11-25 08:00, Richard Purdie via lists.openembedded.org wrote: > Hi Qi, > > On Fri, 2022-11-25 at 05:56 +0000, Chen Qi wrote: >> The AB is actually running 'all' tests. >> >> The '--skip-tests (-R)' option means 'Run all (unhidden) tests except >> the ones specified.', according to its help message. This is also its >> actual effect for now. You can see there's an implicit 'run all >> tests' meaning in this option. >> The '-T (--exclude-tag)' options means 'Exclude all (unhidden) tests >> that match any of the specified tag(s)', according to its help >> message. >> >> So the AB's oe-selftest command means: execute all tests except the >> ones tagged with 'machine' and 'toolchain-user', and also skip >> 'distrodata.Distrodata.test_checkpkg >> buildoptions.SourceMirroring.test_yocto_source_mirror reproducible'. > > You're right, a different build has -t machine and I'm confusing the > two, sorry. > > I don't really like making changes which require lockstep changes to > the autobuilder configuration but I can see why we could do with doing > so here. > > I was thinking there should be symettry between -r and -R like there is > with -T and -t but that also probably doesn't make sense when you think > about it more. > > Alex: We'll probably have to work out how to make this change... Nudge and Happy Holidays. ../Randy > > Cheers, > > Richard > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#173766): https://lists.openembedded.org/g/openembedded-core/message/173766 > Mute This Topic: https://lists.openembedded.org/mt/95231396/3616765 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [randy.macleod@windriver.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On 25/12/2022 19:13:43-0500, Randy MacLeod wrote: > On 2022-11-25 08:00, Richard Purdie via lists.openembedded.org wrote: > > Hi Qi, > > > > On Fri, 2022-11-25 at 05:56 +0000, Chen Qi wrote: > > > The AB is actually running 'all' tests. > > > > > > The '--skip-tests (-R)' option means 'Run all (unhidden) tests except > > > the ones specified.', according to its help message. This is also its > > > actual effect for now. You can see there's an implicit 'run all > > > tests' meaning in this option. > > > The '-T (--exclude-tag)' options means 'Exclude all (unhidden) tests > > > that match any of the specified tag(s)', according to its help > > > message. > > > > > > So the AB's oe-selftest command means: execute all tests except the > > > ones tagged with 'machine' and 'toolchain-user', and also skip > > > 'distrodata.Distrodata.test_checkpkg > > > buildoptions.SourceMirroring.test_yocto_source_mirror reproducible'. > > > > You're right, a different build has -t machine and I'm confusing the > > two, sorry. > > > > I don't really like making changes which require lockstep changes to > > the autobuilder configuration but I can see why we could do with doing > > so here. > > > > I was thinking there should be symettry between -r and -R like there is > > with -T and -t but that also probably doesn't make sense when you think > > about it more. > > > > Alex: We'll probably have to work out how to make this change... > > Nudge and Happy Holidays. Yes, this is still on my radar.
diff --git a/meta/lib/oeqa/selftest/context.py b/meta/lib/oeqa/selftest/context.py index 78c7a467e2..c7dd03ce37 100644 --- a/meta/lib/oeqa/selftest/context.py +++ b/meta/lib/oeqa/selftest/context.py @@ -154,9 +154,6 @@ class OESelftestTestContextExecutor(OETestContextExecutor): group.add_argument('-a', '--run-all-tests', default=False, action="store_true", dest="run_all_tests", help='Run all (unhidden) tests') - group.add_argument('-R', '--skip-tests', required=False, action='store', - nargs='+', dest="skips", default=None, - help='Run all (unhidden) tests except the ones specified. Format should be <module>[.<class>[.<test_method>]]') group.add_argument('-r', '--run-tests', required=False, action='store', nargs='+', dest="run_tests", default=None, help='Select what tests to run (modules, classes or test methods). Format should be: <module>.<class>.<test_method>') @@ -171,6 +168,9 @@ class OESelftestTestContextExecutor(OETestContextExecutor): action="store_true", default=False, help='List all available tests.') + parser.add_argument('-R', '--skip-tests', required=False, action='store', + nargs='+', dest="skips", default=None, + help='Skip the tests specified. Format should be <module>[.<class>[.<test_method>]]') parser.add_argument('-j', '--num-processes', dest='processes', action='store', type=int, help="number of processes to execute in parallel with")
For now, there's no easy way to skip some specified tests when using 'oe-selftest -r'. This is because '-R' and '-r' are in the mutually exclusive group. Looking at the codes, the codes have already allowed running specified tests and skipping specified tests at the same time. So move '-R' out of the mutually exclusive group and change its help message. After this change, the following command could be used. oe-selftest -r A -R A.B.C This does introduce a behavior regression. Previous 'oe-selftest -R xxx' needs to now be changed to 'oe-selftest -a -R xxx'. Signed-off-by: Chen Qi <Qi.Chen@windriver.com> --- meta/lib/oeqa/selftest/context.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)