Re: [Xen-devel] [PATCH XTF 3/3] xtf-runner: regularise runner exit code

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 25/07/16 12:25, Ian Jackson wrote:
> Andrew Cooper writes ("Re: [PATCH XTF 3/3] xtf-runner: regularise runner exit 
> code"):
>> On 22/07/16 10:29, Wei Liu wrote:
>>> This would cause a FAILURE to overwrite previous ERROR result. Is that
>>> what you want?
>> When running more than one test, the overall result should be the most 
>> severe.  So yes, a subsequent FAILURE should override an ERROR.
> "Error" is surely a more severe problem than "Failure".

No.  Error means "something unexpected went wrong", while Failure is
"the test worked, and identified that the area under test is defective".

Error is typically a test case bug or infrastructure issue.

> In particular, Error might mask a failure.

Indeed it could.  Either should cause a prompt investigation and effort
to resolve the issues, but one must be more severe than the other and
this is the way things are specified.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel