[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [XEN PATCH 5/9] CI: Have the gitlab job fail on tools/tests failure
On Tue, Jun 03, 2025 at 02:41:50PM +0100, Andrew Cooper wrote: > On 03/06/2025 1:42 pm, Anthony PERARD wrote: > > if [ -n "$retrieve_xml" ]; then > > nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null > > + # Findout if one of the test failed > > + if ! grep -q '</testsuites>' tests-junit.xml; then > > + echo "ERROR: tests-junit.xml is incomplete or missing." > > + TEST_RESULT=1 > > + elif grep -q '</failure>' tests-junit.xml; then > > + TEST_RESULT=1 > > + fi > > fi > > > > exit "$TEST_RESULT" > > A couple of things. > > From my experimentation with junit, > https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1849342222/test_report?job_name=kbl-xtf-x86-64-gcc-debug > we can also use </error> for classification. I'm also very disappointed > in Gitlab classifying <warning> as success. According to the documentation [1] which point to this junit xml format [2] the only elements (and path) are: testsuites.testsuite.testcase.failure "error" or "warning" don't exist. There's the attributes `type` in <failure> but this isn't explained how it's used. But I guess if we follow the link in [2], go through web.archive.org, we can find [3] which has "skipped", "error", "failure", but still no "warning". [1] https://docs.gitlab.com/ci/testing/unit_test_reports/#unit-test-reporting-workflow [2] https://www.ibm.com/docs/en/developer-for-zos/16.0?topic=formats-junit-xml-format [3] https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd > Not for this patch, but for XTF I need to be able to express "tolerable > failure". (All branches of Xen will run the same tests, and we don't > have OSSTest to deem "fail never passed" as non-blocking.) According to [1], there's a notion of "Existing failures", but that might show up only on merge request. > Even if the job passes overall, I want tolerable failures to show up in > the UI, so I have to use <failure> in junit.xml. But that means needing > to be more selective, and I don't have a good idea of how to do this. > (I have one terrible idea, which is </failure type=tolerable"> which > will escape that grep, but it feels like (ab)buse of XML.) At the moment, `run-tools-tests` write '<failure type="failure"' on failure, so we could grep on that instead event if it is sligtly more fragile. I've choosen to grep on '</failure>' at first because that's much less likely to be written differently, while the attributes in the tag could be written in a different order. Then, we can always use `sed` and extract the "type" to check it: sed -n 's/.*<failure \(\|.* \)type="\([^"]\+\)" .*/\2/p' tests-junit.xml | while read type; do case $type in failure) echo fail;; tolerable) echo ok;; *) echo error unknwon type $type;; esac done But that maybe going a bit too far :-) Cheers, -- Anthony PERARD
|
![]() |
Lists.xenproject.org is hosted with RackSpace, monitoring our |