[Xen-users] xen randomly crashes all VMs hosted on iSCSI NAS array

[VPS Lime <vpslime@xxxxxxxxx>]

I inherited a xen server that is setup to have all the VM images hosted on an iSCSI mounted NAS array.  We been experiencing a random (about every 2-3 days) issue where xen would crash all the VMs, leaving nothing but the Domain0 running.  What appears to be happening is something causes the iSCI mount to hiccup.  Running "vgchange -a y" and restarting all the VMs brings everything up.  Nothing appears to be wrong with the NAS array - there are a dozen other servers attached to it that never have a problem.  The xend log does not have anything useful in it and I'm at a loss to figure out what is causing this.  The only suggestion I've heard is maybe the memory usage is too high and it is causing the box to be unstable.  If anyone has any suggestions or any additional logs I should be looking at, I'd really appreciate it.

Host OS: CentOS 5.5

Xen kernel: xen.gz-2.6.18-194.11.4.el5

iSCSI libraries: iscsi-initiator-utils-6.2.0.871-0.16.el5

Memory on server: 32G

Total memory allocated for VMs running paravirt: 19,384 M

Total memory allocated for VMs running HVM: 2,688 M

Results of xm top:

xentop - 10:11:06   Xen 3.1.2-194.11.4.el5

39 domains: 1 running, 38 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown

Mem: 25165116k total, 25150528k used, 14588k free    CPUs: 8 @ 1995MHz

      NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k) MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR SSID

  Domain-0 -----r 1583   17.1    3220540   12.8   no limit       n/a     8   32     1932    32747    0        0        0        0    0

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users