Re: [Xen-devel] [OSSTEST PATCH v11 16/20] ts-openstack-tempest: Update list of skipped tests

[Anthony PERARD <anthony.perard@xxxxxxxxxx>]

On Fri, Jun 23, 2017 at 02:47:37PM +0100, Ian Jackson wrote:
> Anthony PERARD writes ("[OSSTEST PATCH v11 16/20] ts-openstack-tempest: 
> Update list of skipped tests"):
> > Signed-off-by: Anthony PERARD <anthony.perard@xxxxxxxxxx>
> 
> Again, does this not mean we're going to suffer a maintenance burden
> as tempest grows new inapplicable tests ?

That exactly what is happening with the OpenStack CI loop, from time to
time, there are new tests, that can fail, and diffenrent ways to fix
this (fix the bug, fix a configuration, or just skip the tests).

Recently, we've actually push the list of tests to skip into nova.git,
but I think it is only available in master.

> Other possibilities that come to my mind: ideally the tempest tests
> would have metadata so that particular classes of tests could be
> skipped.  Alternatively, if they aren't disruptive or very slow, if
> run, we could run them anyway as substeps.
> 
> Which makes me think: maybe the tempest tests want to be substeps
> anyway.  Is that possible ?  (Does tempest speak subunit or
> something ?)

I think it is subunit, yes. And I've got the command that is been run by
tempest (as it's printed on stdout).

running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} 
${OS_TEST_PATH:-./tempest/test_discover} --list

then several:
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} 
${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpzSNsrB



-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel