Re: [Xen-devel] [OSSTEST PATCH v11 16/20] ts-openstack-tempest: Update list of skipped tests

[Ian Jackson <ian.jackson@xxxxxxxxxxxxx>]

Anthony PERARD writes ("Re: [OSSTEST PATCH v11 16/20] ts-openstack-tempest: 
Update list of skipped tests"):
> On Fri, Jun 23, 2017 at 02:47:37PM +0100, Ian Jackson wrote:
> > Again, does this not mean we're going to suffer a maintenance burden
> > as tempest grows new inapplicable tests ?
> 
> That exactly what is happening with the OpenStack CI loop, from time to
> time, there are new tests, that can fail, and diffenrent ways to fix
> this (fix the bug, fix a configuration, or just skip the tests).
> 
> Recently, we've actually push the list of tests to skip into nova.git,
> but I think it is only available in master.

Aha.  Well, if the stable branch is stable then the set of tests to
skip there is probably stable too ?  And on master it's built-in ?  So
this actually won't be a problem - in the sense that this will be
approximately the last necessary update to the list of tests to skip ?

> > Which makes me think: maybe the tempest tests want to be substeps
> > anyway.  Is that possible ?  (Does tempest speak subunit or
> > something ?)
> 
> I think it is subunit, yes. And I've got the command that is been run by
> tempest (as it's printed on stdout).
> 
> running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
> OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
> OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
> OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
> ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} 
> ${OS_TEST_PATH:-./tempest/test_discover} --list
> 
> then several:
> running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
> OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
> OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
> OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
> ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} 
> ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpzSNsrB

It would be really good to have those individual subunit results as
substeps.  There is no subunit parser in osstest yet but we could have
one.

What version of subunit does it print out ?  The subunit v1 protocol
is lovely and simple but there is a Second System :-/.

I guess that tempest doesn't stop on the first failed test ?  So
perhaps we could just tolerate the failed-but-not-skipped tests ?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel