[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] GPF Heisenbug with rumprun-xen

On 05/02/15 15:51, Ian Jackson wrote:
It looks to be some sort of Heisenbug in the rump kernel stuff.

I agree.  We had a failure on the 16th of January which looked like
some kind of race:
  (Subject: Re: [Xen-devel] [rumpuserxen test] 33416: regressions - FAIL)

Aha!  I told you I don't believe in cosmic rays ;)

show a history of random failures at the xenstorels step.

At least the ones as far back as 33830 (the last one with logs still
available) all show signs of what looks like memory corruption of some

Thanks for the digging.  (I have left the quoted text in for the
benefit of rumpkernel-users.)

The first failure in that history that looks like part of this is
flight 33690.  We don't have logs for that any more but it used
   rumpuserxen 598ceb54916b
   xen         49de0b57b853
   netbsdsrc   17a547ca2943
Failure probability after then seems about 20%.  If I go back 10
passes from 33690 I get to 33611 which used
   rumpuserxen ffcd777f8062
   xen         0d2879062076
   netbsdsrc   a7c6b12e1752

It seems unlikely that the difference is g>
oing to be due to changes
in the versions of linux, linuxfirmware, ovmf, qemu[u] or seabios.
buildrump.sh has been 47b1a5eef43c throughout.

The diffs for rumpuserxen and netbsdsrc between those revisions are luckily small. I couldn't spot anything in there which would immediately look suspicious. The most suspicious change is calling sched_yield() as part of the bootstrap process, but that's not very dramatic as far as suspicious goes. TLS support was added, but I'm not sure how that would affect threads which do not use TLS. That said, TLS did work right off the bat, so it is a bit suspicious ...

Is it possible that some change in xen is tickling the bug? That would explain why attempts to reproduce the bug in other setups have failed. Is it easy to fire off runs with arbitrary revisions of each repo?

Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.