[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [XEN PATCH 5/9] CI: Have the gitlab job fail on tools/tests failure
On Tue, Jun 03, 2025 at 02:41:50PM +0100, Andrew Cooper wrote: > On 03/06/2025 1:42 pm, Anthony PERARD wrote: > > From: Anthony PERARD <anthony.perard@xxxxxxxxxx> > > > > We can't rely on an exit value from `run-tools-tests` since we only > > have the console output. `console.exp` only look for success or it > > times out. We could parse the console output, but the junit is more > > concise. Also check if we have it or fail as well. > > > > Signed-off-by: Anthony PERARD <anthony.perard@xxxxxxxxxx> > > --- > > automation/scripts/qubes-x86-64.sh | 7 +++++++ > > 1 file changed, 7 insertions(+) > > > > diff --git a/automation/scripts/qubes-x86-64.sh > > b/automation/scripts/qubes-x86-64.sh > > index 046137a4a6..7a4c5ae489 100755 > > --- a/automation/scripts/qubes-x86-64.sh > > +++ b/automation/scripts/qubes-x86-64.sh > > @@ -298,6 +298,13 @@ TEST_RESULT=$? > > > > if [ -n "$retrieve_xml" ]; then > > nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null > > + # Findout if one of the test failed > > + if ! grep -q '</testsuites>' tests-junit.xml; then > > + echo "ERROR: tests-junit.xml is incomplete or missing." > > + TEST_RESULT=1 > > + elif grep -q '</failure>' tests-junit.xml; then > > + TEST_RESULT=1 > > + fi > > fi > > > > exit "$TEST_RESULT" > > A couple of things. > > From my experimentation with junit, > https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1849342222/test_report?job_name=kbl-xtf-x86-64-gcc-debug > we can also use </error> for classification. I'm also very disappointed > in Gitlab classifying <warning> as success. > > Not for this patch, but for XTF I need to be able to express "tolerable > failure". (All branches of Xen will run the same tests, and we don't > have OSSTest to deem "fail never passed" as non-blocking.) > > Even if the job passes overall, I want tolerable failures to show up in > the UI, so I have to use <failure> in junit.xml. But that means needing > to be more selective, and I don't have a good idea of how to do this. > (I have one terrible idea, which is </failure type=tolerable"> which > will escape that grep, but it feels like (ab)buse of XML.) But that automation/ dir (including the run-tools-tests script) is per-branch, so you can specify there which tests should be considered failure and which just warning, no? It will require few more bits in the script, but fundamentally shouldn't be a problem? -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab Attachment:
signature.asc
|
![]() |
Lists.xenproject.org is hosted with RackSpace, monitoring our |