Re: [Xen-devel] [OSSTEST PATCH v11 16/20] ts-openstack-tempest: Update list of skipped tests

[Anthony PERARD <anthony.perard@xxxxxxxxxx>]

On Fri, Jun 23, 2017 at 04:38:05PM +0100, Ian Jackson wrote:
> Anthony PERARD writes ("Re: [OSSTEST PATCH v11 16/20] ts-openstack-tempest: 
> Update list of skipped tests"):
> > On Fri, Jun 23, 2017 at 02:47:37PM +0100, Ian Jackson wrote:
> > > Again, does this not mean we're going to suffer a maintenance burden
> > > as tempest grows new inapplicable tests ?
> > 
> > That exactly what is happening with the OpenStack CI loop, from time to
> > time, there are new tests, that can fail, and diffenrent ways to fix
> > this (fix the bug, fix a configuration, or just skip the tests).
> > 
> > Recently, we've actually push the list of tests to skip into nova.git,
> > but I think it is only available in master.
> 
> Aha.  Well, if the stable branch is stable then the set of tests to
> skip there is probably stable too ?  And on master it's built-in ?  So
> this actually won't be a problem - in the sense that this will be
> approximately the last necessary update to the list of tests to skip ?

The built-in list is to be consume by the CI, it is not used
automatically. It a bash script to be sourced. I have not yet try to
have osstest use it. That will be for later.

> > > Which makes me think: maybe the tempest tests want to be substeps
> > > anyway.  Is that possible ?  (Does tempest speak subunit or
> > > something ?)
> > 
> > I think it is subunit, yes. And I've got the command that is been run by
> > tempest (as it's printed on stdout).
> > 
> > running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
> > OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
> > OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
> > OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
> > ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} 
> > ${OS_TEST_PATH:-./tempest/test_discover} --list
> > 
> > then several:
> > running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
> > OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
> > OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
> > OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
> > ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} 
> > ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpzSNsrB
> 
> It would be really good to have those individual subunit results as
> substeps.  There is no subunit parser in osstest yet but we could have
> one.
> 
> What version of subunit does it print out ?  The subunit v1 protocol
> is lovely and simple but there is a Second System :-/.

Right now, tempest print on stdout something much simpler, lines liked
this:
{1} 
tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario
 [71.009969s] ... ok
Followed by summary of all test, and the output of failed tests.

But it can be told to print subunit v2, which can be pipped to a different
program to transform into want we want, including simple cvs, subunit v1, and
other.

I thing we could try to parse subunit v1, grab the start and finish time
of a test, the result, and the output of the tests. And have all that as
subunit. During my manual tests, `tempest run --subunit ... | subunit-2to1`
only print the stuff about a tests once it is finish, so I guess we'll
have to rewrite substep_start() (or another function) to take time as
parameter.

> I guess that tempest doesn't stop on the first failed test ?  So
> perhaps we could just tolerate the failed-but-not-skipped tests ?

It does not, it run everything, no matter how many test fails.

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel