[Xen-API] [XCP 1.1] [BUG] Hangs during migration

[George Shuklin <shuklin@xxxxxxxxxxx>]

Good day.

I've finally got 'last breath' of system during hangs.

System: XCP 1.1

CPU:
vendor_id    : GenuineIntel
cpu family    : 6
model        : 26
model name    : Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz
stepping    : 5

That hang usually happens during mass migration from host (e.g. host 
evacuate). Host become unresponsive (no console responce, no management 
interface pings, no ping to VM's (different network)).
Here trace from netconsole before (or during) hang process:

 Sep 24 00:14:23 device vif132.0 left promiscuous mode
 Sep 24 00:14:38 device vif133.0 left promiscuous mode
 Sep 24 00:14:54 device vif134.0 left promiscuous mode
 Sep 24 00:16:56 SRV BUG: soft lockup - CPU#2 stuck for 61s! 
[selecon-host:7812]
 Sep 24 00:16:56 Modules linked in:
 Sep 24 00:16:56 SRV  iscsi_tcp
 Sep 24 00:16:56 SRV  libiscsi_tcp
 Sep 24 00:16:56 SRV  libiscsi
 Sep 24 00:16:56 SRV  scsi_transport_iscsi
 Sep 24 00:16:56 SRV  netconsole
 Sep 24 00:16:56 SRV  nfs
 Sep 24 00:16:56 SRV  nfs_acl
 Sep 24 00:16:56 SRV  auth_rpcgss
 Sep 24 00:16:56 SRV  lockd
 Sep 24 00:16:56 SRV  sunrpc
 Sep 24 00:16:56 SRV  openvswitch_mod
 Sep 24 00:16:56 SRV  battery
 Sep 24 00:16:56 SRV  serial_core
 Sep 24 00:16:56 SRV  tpm_tis
 Sep 24 00:16:56 SRV  tpm
 Sep 24 00:16:56 SRV  tpm_bios
 Sep 24 00:16:56 SRV  i2c_i801
 Sep 24 00:16:56 SRV  i2c_core
 Sep 24 00:16:56 SRV  pcspkr
 Sep 24 00:16:56 SRV  dm_region_hash
 Sep 24 00:16:56 SRV  dm_log
 Sep 24 00:16:56 SRV  dm_mod
 Sep 24 00:16:56 SRV  ide_gd_mod
 Sep 24 00:16:56 SRV  ahci
 Sep 24 00:16:56 SRV  libata
 Sep 24 00:16:56 SRV CR0: 80050033 CR2: f18e7004 CR3: 31148000 CR4: 
00002660
 Sep 24 00:16:56 SRV DR6: ffff0ff0 DR7: 00000400
 Sep 24 00:16:56 Call Trace:
 Sep 24 00:16:56 SRV  [<c02bc928>] ? privcmd_ioctl+0xa8/0x1060
 Sep 24 00:16:56 SRV  [<c037d4c9>] ? schedule+0x2e9/0x970
 Sep 24 00:16:56 SRV  [<c0128490>] ? check_preempt_wakeup+0xe0/0x1d0
 Sep 24 00:16:56 SRV  [<c014aa3e>] ? futex_wait_queue_me+0xae/0xe0
 Sep 24 00:16:56 SRV  [<c014a90a>] ? futex_wait_setup+0x9a/0x120
 Sep 24 00:16:56 SRV  [<c014abd7>] ? futex_wait+0x167/0x230
 Sep 24 00:16:56 SRV  [<c0106a51>] ? local_clock+0x41/0x80
 Sep 24 00:16:56 SRV  [<c0106b60>] ? xen_clocksource_read+0x20/0x130
 Sep 24 00:16:56 SRV  [<c02bc880>] ? privcmd_ioctl+0x0/0x1060
 Sep 24 00:16:56 SRV  [<c01d3e26>] proc_reg_unlocked_ioctl+0x66/0x100
 Sep 24 00:16:56 SRV  [<c01d3dc0>] ? proc_reg_unlocked_ioctl+0x0/0x100
 Sep 24 00:16:56 SRV  [<c01a0768>] vfs_ioctl+0x28/0x80
 Sep 24 00:16:56 SRV  [<c0145378>] ? sched_clock_cpu+0x28/0x160
 Sep 24 00:16:56 SRV  [<c01a0915>] do_vfs_ioctl+0x85/0x580
 Sep 24 00:16:56 SRV  [<c014340c>] ? __remove_hrtimer+0x2c/0x50
 Sep 24 00:16:56 SRV  [<c014387a>] ? hrtimer_try_to_cancel+0x3a/0x80
 Sep 24 00:16:56 SRV  [<c013fea9>] ? sys_timer_settime+0x269/0x2a0
 Sep 24 00:16:56 SRV  [<c014d0d6>] ? sys_futex+0x66/0x120
 Sep 24 00:16:56 SRV  [<c01a0e70>] sys_ioctl+0x60/0x70
 Sep 24 00:16:56 SRV  [<c01044e1>] syscall_call+0x7/0xb
 Sep 24 00:18:02 SRV BUG: soft lockup - CPU#1 stuck for 61s! 
[domainWatcher:8789]
 Sep 24 00:18:02 Modules linked in:
 Sep 24 00:18:02 SRV  iscsi_tcp
 Sep 24 00:18:02 SRV  libiscsi_tcp
 Sep 24 00:18:02 SRV  libiscsi
 Sep 24 00:18:02 SRV  scsi_transport_iscsi
 Sep 24 00:18:02 SRV  netconsole
 Sep 24 00:18:02 SRV  nfs
 Sep 24 00:18:02 SRV  nfs_acl
 Sep 24 00:18:02 SRV  auth_rpcgss
 Sep 24 00:18:02 SRV  lockd
 Sep 24 00:18:02 SRV  sunrpc
 Sep 24 00:18:02 SRV  openvswitch_mod
 Sep 24 00:18:02 SRV  [<c014340c>] ? __remove_hrtimer+0x2c/0x50
 Sep 24 00:18:02 SRV  [<c01044e1>] syscall_call+0x7/0xb
 Sep 24 00:18:02 SRV BUG: soft lockup - CPU#2 stuck for 61s! 
[selecon-host:7812]
 Sep 24 00:18:02 Modules linked in:
 Sep 24 00:18:02 SRV  iscsi_tcp
 Sep 24 00:18:02 SRV  libiscsi_tcp
 Sep 24 00:18:02 SRV  libiscsi
 Sep 24 00:18:02 SRV  scsi_transport_iscsi
 Sep 24 00:18:02 SRV  netconsole
 Sep 24 00:18:02 SRV  nfs
 Sep 24 00:18:02 SRV  nfs_acl
 Sep 24 00:18:02 SRV  auth_rpcgss
 Sep 24 00:18:02 SRV  lockd
 Sep 24 00:18:02 SRV  sunrpc
 Sep 24 00:18:02 SRV  openvswitch_mod


_______________________________________________
Xen-api mailing list
Xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api