[Xen-devel] OVMF related osstest failures on multiple branches

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

These two tests are failing all over the place at the moment:
    test-amd64-i386-xl-qemuu-ovmf-amd64.debian-hvm-install
    test-amd64-amd64-xl-qemuu-ovmf-amd64.debian-hvm-install

For the rest of this mail I will only consider the test-amd64-amd64-* one,
assuming they are the same.

The issues appear to have started on 18/12 last year.

Affected branches seem to be:
    xen-4.6-testing, xen-4.5-testing, xen-4.4-testing
    qemu-upstream-4.6-testing, qemu-upstream-4.5-testing, 
qemu-upstream-4.4-testing

xen-unstable appears to be unaffected.

qemu-mainline is blocked by a build failure.

qemu-upstream-unstable is OK, it had a pass on 18/12, so it's possible that
if the issue is elsewhere this tree simply hasn't seen it yet.

(links to all branch histories are below for convenience)

All branches have had a pass with ovmf revision cb9a7ebabcd6 and are still
using that version now during the failures. I don't think this is an
ovmf.git issue.

In all trees multiple revisions changed over the "event horizon" (i.e. when
the failures occurred), the following summarises what changed for each:

                pass/fail host          osstest linux   qemuu   xen
xen 4.6         godello1/rimava0        Y[0]    N[4]    N[6]    Y[8]
xen 4.5         pinot0/huxelrebe1       Y[0]    N[4]    N[6]    Y[8]
xen 4.4         rimava1/italia0         Y[0]    N[4]    N[6]    Y[9]    

qemu 4.6        rimava1/baroque1        Y[1]    Y[5]    Y[7]    Y[10]
qemu 4.5[11]    chardonay0/chardonay1   Y[2]    Y[5]    Y[7]    Y[10]
qemu 4.4        elbling1/huxelrebe1     Y[3]    Y[5]    Y[7]    Y

[0] osstest: af4a02e671de..f610ea162836 == update to Jessie
[1]Âosstest: 54f237784d4b..f610ea162836
[2]Âosstest: f795781327c0..f610ea162836
[3] osstest: 3cd8b12a4fb1..f610ea162836
[4] Linux 3.14.58
[5] Linux updated from various versions (54, 50, 54) to 3.14.58
[6] qemuu 980dfcf5b8d8
[7] Two XSA-155 fixes
[8] qemut updated for XSA-155.
[9] Fixes for XSA-165 and XSA-166
[10] Lots of changes, from September/October 2015 until now, not includingÂ
  ÂXSA-155, -165 or -166.
[11] Flight before event horizon is 62414, which failed a later step, butÂ
  Âis considered a pass of the debian-hvm-install step.

The hosts also all changed, but there is no apparent pattern to the failing
hosts.

Given that Xen * passes with Linux 3.14.58 I doubt the change of Linux in
the QEMU flights to use that version is the issue.

Given that QEMU didn't change in Xen * it seems unlikely that it is the
XSA-155 fixes in the QEMU flights which are causing this.

Given that the xen changes in the Xen * branches all post date the version
of Xen ones in the QEMU * branches, I doubt it is those (especially the
qemut only changes).

Which basically only leaves the switch to using a Debian Jessie build and
runtime environment. As well as the host side stuff we also change the
Debian HVM ISO from Wheezy to Jessie. The fact that none of the bisects
have made any progress (Xen * failed to repro basis pass, QEMU * didn't
seem to find a basis pass to start with) seems to support this.

The fly in the ointment of this theory is that the Jessie upgrade doesn't
seem to have broken unstable in this way, for some reason, which is what I
intend to investigate first.

Ian.

Job histories:

http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-qemuu-ovmf-amd64/xen-4.6-testing.html
http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-qemuu-ovmf-amd64/xen-4.5-testing.html
http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-qemuu-ovmf-amd64/xen-4.4-testing.html

http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-qemuu-ovmf-amd64/qemu-upstream-4.6-testing.html
http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-qemuu-ovmf-amd64/qemu-upstream-4.5-testing.html
http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-qemuu-ovmf-amd64/qemu-upstream-4.4-testing.html

Bisects:
http://logs.test-lab.xenproject.org/osstest/results/bisect/xen-4.6-testing/test-amd64-amd64-xl-qemuu-ovmf-amd64.debian-hvm-install.html
http://logs.test-lab.xenproject.org/osstest/results/bisect/xen-4.5-testing/test-amd64-amd64-xl-qemuu-ovmf-amd64.debian-hvm-install.html
(4.4 hasn't tackled this one yet)

http://logs.test-lab.xenproject.org/osstest/results/bisect/qemu-upstream-4.6-testing/test-amd64-amd64-xl-qemuu-ovmf-amd64.debian-hvm-install.html
http://logs.test-lab.xenproject.org/osstest/results/bisect/qemu-upstream-4.5-testing/test-amd64-amd64-xl-qemuu-ovmf-amd64.debian-hvm-install.html
http://logs.test-lab.xenproject.org/osstest/results/bisect/qemu-upstream-4.4-testing/test-amd64-amd64-xl-qemuu-ovmf-amd64.debian-hvm-install.html


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel