[Xen-users] BUG: soft lockup - CPU#6 stuck for 61s! [apache2:18704]

[Sergi Seira <s.seira@xxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

there has been some threads about this topic before, but no workaround so far 
has worked for me (clocksource jiffies, pit, ...)

I'm running debian squeeze 2.6.32-5-xen-amd64.

DomU usually have 6 or 7 cores and 6 or 7 GB of RAM.

Only happens on servers with considerable load and memory usage.

This is the the message that once starts showing degradation begins until load 
reaches impossible values and I have to destroy the domU because soft lockup 
makes processes uninterruptible :

[ 1957.432378] BUG: soft lockup - CPU#6 stuck for 61s! [apache2:18704]
[ 1957.432378] Modules linked in: quota_v2 quota_tree ip6table_filter 
ip6_tables act_police cls_flow cls_fw cls_u32 sch_htb sch_hfsc sch_ingress 
sch_sfq xt_time xt_connlimit xt_realm iptable_raw
xt_comment xt_recent xt_policy ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP 
ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah ipt_addrtype nf_nat_tftp 
nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp 
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip 
nf_conntrack_proto_sctp nf_conntrack_pptp
nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns 
nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core 
xt_tcpmss xt_pkttype xt_physdev xt_owner xt_NFQUEUE
xt_NFLOG nfnetlink_log xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length 
xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack 
xt_CONNMARK xt_connmark xt_CLASSIFY ipt_LOG
xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 
nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables x_tables snd_pcm 
snd_timer snd soundcore snd_page_alloc pcspkr
evdev ext4 mbcache jbd2 crc16 dm_mod raid1 md_mod xen_netfront xen_blkfront
[ 1957.432378] CPU 6:
[ 1957.432378] Modules linked in: quota_v2 quota_tree ip6table_filter 
ip6_tables act_police cls_flow cls_fw cls_u32 sch_htb sch_hfsc sch_ingress 
sch_sfq xt_time xt_connlimit xt_realm iptable_raw
xt_comment xt_recent xt_policy ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP 
ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah ipt_addrtype nf_nat_tftp 
nf_nat_snmp_basic nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp 
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip 
nf_conntrack_proto_sctp nf_conntrack_pptp
nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns 
nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core 
xt_tcpmss xt_pkttype xt_physdev xt_owner xt_NFQUEUE
xt_NFLOG nfnetlink_log xt_multiport xt_MARK xt_mark xt_mac xt_limit xt_length 
xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack 
xt_CONNMARK xt_connmark xt_CLASSIFY ipt_LOG
xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 
nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables x_tables snd_pcm 
snd_timer snd soundcore snd_page_alloc pcspkr
evdev ext4 mbcache jbd2 crc16 dm_mod raid1 md_mod xen_netfront xen_blkfront
[ 1957.432378] Pid: 18704, comm: apache2 Tainted: G      D    
2.6.32-5-xen-amd64 #1
[ 1957.432378] RIP: e030:[<ffffffff8130d116>]  [<ffffffff8130d116>] 
_spin_lock+0x13/0x1b
[ 1957.432378] RSP: e02b:ffff88018c249b10  EFLAGS: 00000297
[ 1957.432378] RAX: 00000000000000d2 RBX: ffff88018c249b28 RCX: ffff88018c249b68
[ 1957.432378] RDX: 00000000000000d1 RSI: 0000000000000200 RDI: ffff8801b8c49580
[ 1957.432378] RBP: 0000000000000200 R08: 0000000000000008 R09: ffffffff814eb900
[ 1957.432378] R10: ffff8801bf373880 R11: ffffffff811ba500 R12: ffff8801b8c49500
[ 1957.432378] R13: 000000000000c580 R14: ffff88018c249b28 R15: ffffffff814eb8c0
[ 1957.432378] FS:  00007fe81d353740(0000) GS:ffff88000aa4c000(0000) 
knlGS:0000000000000000
[ 1957.432378] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1957.432378] CR2: 00007fe820830038 CR3: 0000000001001000 CR4: 0000000000000660
[ 1957.432378] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1957.432378] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1957.432378] Call Trace:
[ 1957.432378]  [<ffffffff8100dd87>] ? xen_exit_mmap+0xf8/0x136
[ 1957.432378]  [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001
[ 1957.432378]  [<ffffffff810d1464>] ? exit_mmap+0x5a/0x148
[ 1957.432378]  [<ffffffff8100ecf2>] ? check_events+0x12/0x20
[ 1957.432378]  [<ffffffff811ba500>] ? dummycon_dummy+0x0/0x3
[ 1957.432378]  [<ffffffff8104cc7d>] ? mmput+0x3c/0xdf
[ 1957.432378]  [<ffffffff81050882>] ? exit_mm+0x102/0x10d
[ 1957.432378]  [<ffffffff8100ec99>] ? xen_irq_enable_direct_end+0x0/0x7
[ 1957.432378]  [<ffffffff810522a7>] ? do_exit+0x1f8/0x6c6
[ 1957.432378]  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1
[ 1957.432378]  [<ffffffff8130d13a>] ? _spin_unlock_irqrestore+0xd/0xe
[ 1957.432378]  [<ffffffff8104f523>] ? release_console_sem+0x17e/0x1af
[ 1957.432378]  [<ffffffff8130dfdd>] ? oops_end+0xaf/0xb4
[ 1957.432378]  [<ffffffff810135f0>] ? do_invalid_op+0x8b/0x95
[ 1957.432378]  [<ffffffff8100c694>] ? pin_pagetable_pfn+0x2d/0x36
[ 1957.432378]  [<ffffffff810bb157>] ? __alloc_pages_nodemask+0x11c/0x5f5
[ 1957.432378]  [<ffffffff8101293b>] ? invalid_op+0x1b/0x20
[ 1957.432378]  [<ffffffff8100c694>] ? pin_pagetable_pfn+0x2d/0x36
[ 1957.432378]  [<ffffffff8100c690>] ? pin_pagetable_pfn+0x29/0x36
[ 1957.432378]  [<ffffffff810cd73e>] ? __pte_alloc+0x6b/0xc6
[ 1957.432378]  [<ffffffff810cb5f0>] ? pmd_alloc+0x28/0x5b
[ 1957.432378]  [<ffffffff810cd867>] ? handle_mm_fault+0xce/0x80f
[ 1957.432378]  [<ffffffff810d2e28>] ? do_brk+0x227/0x301
[ 1957.432378]  [<ffffffff8130f616>] ? do_page_fault+0x2e0/0x2fc
[ 1957.432378]  [<ffffffff8130d4b5>] ? page_fault+0x25/0x30


Any other possible workaround besides upgrading?

Thanks,
regards,
Sergi
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPV6ezAAoJEN00VGSIizZ0Qh0IAI+44RDVQQJjamJWUtzstyyh
f+OY71nhG/w/XisRZnTVsYv8qaqsIh8FyT2uOEp4Y/Syv/DFxt7X9ey/KKNfqEDl
bpOjoOvIA8+eZhGKCw7hEsEP4H3QEyFSOVwRO4hS5HhsPDdMOA3UTjae1x8o/mcj
DjI97vLZQqDdYrXyXaoAvrUxLHjF8uJCNwQ7JDSdX3OW5wp7KLf15zEWHoHoWaen
wfU8Bor9mqS9E5x3GcV/HTTeqBKLu8psiNIIQ2WVSZ/vF/a7FJo/eKA7kwz7YTQR
rmorRuO58t71sTHNTQG2ISZ+KMGSBNt8HfSQH+OzuDdrmZItwaFN3Vz8YzH5l90=
=Nlkr
-----END PGP SIGNATURE-----

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users