[Xen-devel] linux-3.4 broken on chardonnay and huxelrebe (Re: [linux-3.4 test] 61301: regressions - FAIL)

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

On Fri, 2015-09-04 at 08:58 +0000, osstest service owner wrote:
> flight 61301 linux-3.4 real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/61301/
> 
> Regressions :-(
> 
> Tests which did not succeed and are blocking,
> including tests which could not be run:
>  test-amd64-amd64-xl-qemut-win7-amd64  6 xen-boot          fail REGR. vs. 
> 30511

linux-3.4 seems to have trouble booting on chardonnay and huxelrebe, and
has accumulated _most_ of its jobs in persistent failures on one or the
other of those hosts[1]

Skimming a couple of logs it seems the systems fail for different reasons.

chardonanys are failing to find their root disk while the huxelrebes appear
to boot ok but not get any networking.

Of the chardonnay failures
http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-qemut-win7-amd64/linux-3.4.html
is interesting. All the others show a history of passing on other machines
and then eventually hitting chardonnay and beginning to fail, while 
test-amd64-amd64-xl-qemut-win7-amd64 was booting on chardonnay and then at
some point started failing reliably (with one random looking boot early
on). This might provide sufficient points in the sand to try and bisection.

For the huxelrebes no such luck, I only looked at the first dozen failures
but they all have either always run (and failed) on huxelrebe or they
started failing as soon as they were scheduled on huxelrebe.

However taking test-amd64-amd64-xl-xsm it seems to have passed on huxelrebe
on linux-3.10, which gives us _something_ to go on.

So, I think I can try and convince the bisector to have another go at the
point in time where chardonnay started failing 
test-amd64-amd64-xl-qemut-win7-amd64.

For huxelrebe I can try running some adhoc tests at various points between
3.4 and 3.10 (initially 3.{5,6,7,8,9}) and then the -rcs around the change,
which might point to a region fix. If not then what I really want to do is
an inverted bisect to find the point at which it was fixed...

Ian.

[1]
> select steps.job,steps.testid,steps.status,runvars.val from steps,runvars 
> where steps.flight=runvars.flight and steps.job=runvars.job and 
> steps.flight='61301' and steps.testid = 'xen-boot' and runvars.name like 
> '%host' order by runvars.val;

                          job                          |  testid  | status |    
 val     
-------------------------------------------------------+----------+--------+-------------
 test-amd64-amd64-libvirt-qcow2                        | xen-boot | fail   | 
chardonnay0
 test-amd64-amd64-pygrub                               | xen-boot | fail   | 
chardonnay0
 test-amd64-amd64-libvirt                              | xen-boot | fail   | 
chardonnay0
 test-amd64-amd64-xl-qcow2                             | xen-boot | fail   | 
chardonnay0
 test-amd64-amd64-xl-rtds                              | xen-boot | fail   | 
chardonnay1
 test-amd64-amd64-xl-qemut-win7-amd64                  | xen-boot | fail   | 
chardonnay1
 test-amd64-i386-libvirt-vhd                           | xen-boot | pass   | 
elbling0
 test-amd64-i386-libvirt                               | xen-boot | pass   | 
elbling0
 test-amd64-i386-xl-qcow2                              | xen-boot | pass   | 
elbling0
 test-amd64-amd64-xl-qemuu-win7-amd64                  | xen-boot | pass   | 
elbling1
 test-amd64-i386-xl-qemut-win7-amd64                   | xen-boot | pass   | 
elbling1
 test-amd64-amd64-xl-vhd                               | xen-boot | pass   | 
fiano0
 test-amd64-i386-libvirt-qcow2                         | xen-boot | pass   | 
fiano0
 test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm     | xen-boot | pass   | 
fiano1
 test-amd64-i386-xl-qemuu-win7-amd64                   | xen-boot | pass   | 
fiano1
 test-amd64-amd64-xl-pvh-intel                         | xen-boot | pass   | 
fiano1
 test-amd64-amd64-libvirt-raw                          | xen-boot | pass   | 
godello0
 test-amd64-amd64-libvirt-vhd                          | xen-boot | pass   | 
godello1
 test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm | xen-boot | fail   | 
huxelrebe0
 test-amd64-amd64-xl-credit2                           | xen-boot | fail   | 
huxelrebe0
 test-amd64-amd64-xl-qemut-winxpsp3                    | xen-boot | fail   | 
huxelrebe0
 test-amd64-amd64-xl                                   | xen-boot | fail   | 
huxelrebe0
 test-amd64-i386-xl-qemuu-winxpsp3-vcpus1              | xen-boot | fail   | 
huxelrebe0
 test-amd64-amd64-xl-qemuu-ovmf-amd64                  | xen-boot | fail   | 
huxelrebe0
 test-amd64-amd64-rumpuserxen-amd64                    | xen-boot | fail   | 
huxelrebe0
 test-amd64-i386-xl-qemut-winxpsp3                     | xen-boot | fail   | 
huxelrebe0
 test-amd64-i386-rumpuserxen-i386                      | xen-boot | fail   | 
huxelrebe0
 test-amd64-i386-xl-qemuu-winxpsp3                     | xen-boot | fail   | 
huxelrebe0
 test-amd64-amd64-xl-xsm                               | xen-boot | fail   | 
huxelrebe0
 test-amd64-i386-freebsd10-i386                        | xen-boot | fail   | 
huxelrebe0
 test-amd64-i386-qemut-rhel6hvm-intel                  | xen-boot | fail   | 
huxelrebe0
 test-amd64-i386-xl-qemut-debianhvm-amd64              | xen-boot | fail   | 
huxelrebe0
 test-amd64-amd64-xl-multivcpu                         | xen-boot | fail   | 
huxelrebe0
 test-amd64-i386-libvirt-xsm                           | xen-boot | fail   | 
huxelrebe0
 test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm         | xen-boot | fail   | 
huxelrebe0
 test-amd64-i386-xl-xsm                                | xen-boot | fail   | 
huxelrebe1
 test-amd64-amd64-i386-pvgrub                          | xen-boot | fail   | 
huxelrebe1
 test-amd64-amd64-xl-qemut-debianhvm-amd64             | xen-boot | fail   | 
huxelrebe1
 test-amd64-amd64-xl-qemuu-debianhvm-amd64             | xen-boot | fail   | 
huxelrebe1
 test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm         | xen-boot | fail   | 
huxelrebe1
 test-amd64-amd64-xl-qemuu-winxpsp3                    | xen-boot | fail   | 
huxelrebe1
 test-amd64-i386-freebsd10-amd64                       | xen-boot | fail   | 
huxelrebe1
 test-amd64-i386-qemuu-rhel6hvm-intel                  | xen-boot | fail   | 
huxelrebe1
 test-amd64-i386-xl                                    | xen-boot | fail   | 
huxelrebe1
 test-amd64-i386-xl-qemut-debianhvm-amd64-xsm          | xen-boot | fail   | 
huxelrebe1
 test-amd64-i386-xl-qemut-winxpsp3-vcpus1              | xen-boot | fail   | 
huxelrebe1
 test-amd64-i386-xl-qemuu-debianhvm-amd64              | xen-boot | fail   | 
huxelrebe1
 test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm          | xen-boot | fail   | 
huxelrebe1
 test-amd64-i386-xl-qemuu-ovmf-amd64                   | xen-boot | fail   | 
huxelrebe1
 test-amd64-amd64-amd64-pvgrub                         | xen-boot | pass   | 
italia0
 test-amd64-i386-libvirt-raw                           | xen-boot | pass   | 
italia0
 test-amd64-amd64-libvirt-xsm                          | xen-boot | pass   | 
merlot0
 test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm  | xen-boot | pass   | 
merlot0
 test-amd64-i386-qemuu-rhel6hvm-amd                    | xen-boot | pass   | 
merlot0
 test-amd64-i386-qemut-rhel6hvm-amd                    | xen-boot | pass   | 
merlot1
 test-amd64-i386-xl-raw                                | xen-boot | pass   | 
merlot1
 test-amd64-i386-xl-vhd                                | xen-boot | pass   | 
pinot0
 test-amd64-amd64-xl-pvh-amd                           | xen-boot | pass   | 
pinot1
 test-amd64-amd64-xl-raw                               | xen-boot | pass   | 
rimava1
 test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm    | xen-boot | pass   | 
rimava1
(60 rows)



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel