Re: [XEN PATCH 5/9] CI: Have the gitlab job fail on tools/tests failure

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 03/06/2025 4:58 pm, Anthony PERARD wrote:
> On Tue, Jun 03, 2025 at 02:41:50PM +0100, Andrew Cooper wrote:
>> On 03/06/2025 1:42 pm, Anthony PERARD wrote:
>>>  if [ -n "$retrieve_xml" ]; then
>>>      nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null
>>> +    # Findout if one of the test failed
>>> +    if ! grep -q '</testsuites>' tests-junit.xml; then
>>> +        echo "ERROR: tests-junit.xml is incomplete or missing."
>>> +        TEST_RESULT=1
>>> +    elif grep -q '</failure>' tests-junit.xml; then
>>> +        TEST_RESULT=1
>>> +    fi
>>>  fi
>>>  
>>>  exit "$TEST_RESULT"
>> A couple of things.
>>
>> From my experimentation with junit,
>> https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1849342222/test_report?job_name=kbl-xtf-x86-64-gcc-debug
>> we can also use </error> for classification.  I'm also very disappointed
>> in Gitlab classifying <warning> as success.
> According to the documentation [1] which point to this junit xml format [2]
> the only elements (and path) are:
>     testsuites.testsuite.testcase.failure
> "error" or "warning" don't exist.
> There's the attributes `type` in <failure> but this isn't explained how
> it's used.
>
> But I guess if we follow the link in [2], go through web.archive.org, we
> can find [3] which has "skipped", "error", "failure", but still no
> "warning".
>
>
> [1] 
> https://docs.gitlab.com/ci/testing/unit_test_reports/#unit-test-reporting-workflow
> [2] 
> https://www.ibm.com/docs/en/developer-for-zos/16.0?topic=formats-junit-xml-format
> [3] https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd

I was also very disappointed with the documentation, which seems to be
almost non-existent.

But after some source diving:
https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/lib/gitlab/ci/parsers/test/junit.rb#L69-83

So anything which isn't recognised is deemed to be success.  Lovely :(

I'm also still none the wiser as to why Skip's system_out is a formatted
json blob, unlike the others.


>
>
>> Not for this patch, but for XTF I need to be able to express "tolerable
>> failure".  (All branches of Xen will run the same tests, and we don't
>> have OSSTest to deem "fail never passed" as non-blocking.)
> According to [1], there's a notion of "Existing failures", but that
> might show up only on merge request.
>
>> Even if the job passes overall, I want tolerable failures to show up in
>> the UI, so I have to use <failure> in junit.xml.  But that means needing
>> to be more selective, and I don't have a good idea of how to do this. 
>> (I have one terrible idea, which is </failure type=tolerable"> which
>> will escape that grep, but it feels like (ab)buse of XML.)
> At the moment, `run-tools-tests` write '<failure type="failure"' on
> failure, so we could grep on that instead event if it is sligtly more
> fragile. I've choosen to grep on '</failure>' at first because that's
> much less likely to be written differently, while the attributes in the
> tag could be written in a different order.
>
> Then, we can always use `sed` and extract the "type" to check it:
> sed -n 's/.*<failure \(\|.* \)type="\([^"]\+\)" .*/\2/p' tests-junit.xml | 
> while read type; do
>   case $type in
>     failure) echo fail;;
>     tolerable) echo ok;;
>     *) echo error unknwon type $type;;
>   esac
> done
> But that maybe going a bit too far :-)

There's xq which might be able to do this more nicely, and is packaged
in alpine.

~Andrew