Re: [Xen-devel] [PATCH v1 5/7] ts-livepatch: Initial test-cases.

[Ian Jackson <ian.jackson@xxxxxxxxxxxxx>]

Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] [PATCH v1 5/7] ts-livepatch: 
Initial test-cases."):
> Let me answer only to some as I am fixing the rest per your comments:
> 
> On Thu, Nov 17, 2016 at 12:21:58PM +0000, Ian Jackson wrote:
> > Is it ever possible to continue with the rest of the livepatch tests
> > after one of these command invocations has failed, or does it leave
> > the system in an undefined state ?  If it _is_ possible then it would
> 
> If the invocations have failed that could mean: 
>  - We don't have livepatching enabled (somebody swapped the hypervisor?)
>  - The livepatching did not work right and we have code in the
>    hypervisor that patched something else. Undefined results.
>  - It may be that the test-case failed to load due to dependencies
>    issues - and while that means the system can recover - the test
>    should fail immediately.

Right.

> > be possible to provide per-step results to osstest, but that's only
> > valuable if this would tell us something meaningfully interesting in
> > terms of regressions - eg if we might tolerate a regression of one
> > step but still care about the others.
> 
> At this stage I believe all of these test-cases should work. If they
> don't we got big problems.

OK.  So I think in that case your script should probably fail
completely, and not run any of the further test cases, if any test
fails.

> > > + {cmd => "xen-livepatch revert xen_hello_world", rc => 256},
> > 
> > libxl exit statuses are not very good and should not be relied on,
> > really.  Instead, you should arrange to:
> 
> But this is not libxl. The error values are very much defined
> by xen-livepatch.

No, they clearly aren't.  I just looked at the code and main()
returns like this:
    return !!ret;

So that means that the exit status is either 0 meaning "all is good"
or 1 meaning "something, which might be anything at all, went wrong".

Tests should be arranged to figure out whether the test fails in the
expected way.  With programs whose exit status does not discriminate
between causes of failure, this has to be done by grepping error
messages :-/.

> >   * Run the command with LC_MESSAGES=C
> 
> Aye.
> 
> >   * Fail the test if the exit status is zero
> 
> But some of the test-cases are suppose to return 0 - as the program
> executed correctly.

I meant, if you expect the test to fail, you should insist that it
fails.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel