Re: [Xen-users] BUG: soft lockup - CPU#6 stuck for 61s! [apache2:18704]

[Sergi Seira <s.seira@xxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

well, if someone runs into this bug, i've managed to control it by lowering the 
memory fingerprint of some processes, like apache.
It seems that the less commited memory the less soft lockups under heavy load.

Regards,
Sergi

On 03/07/2012 07:23 PM, Sergi Seira wrote:
> Hello,
> 
> there has been some threads about this topic before, but no workaround so far 
> has worked for me (clocksource jiffies, pit, ...)
> 
> I'm running debian squeeze 2.6.32-5-xen-amd64.
> 
> DomU usually have 6 or 7 cores and 6 or 7 GB of RAM.
> 
> Only happens on servers with considerable load and memory usage.
> 
> This is the the message that once starts showing degradation begins until 
> load reaches impossible values and I have to destroy the domU because soft 
> lockup makes processes uninterruptible :
> 
> [ 1957.432378] BUG: soft lockup - CPU#6 stuck for 61s! [apache2:18704] [ 
> 1957.432378] Modules linked in: quota_v2 quota_tree ip6table_filter 
> ip6_tables act_police cls_flow cls_fw cls_u32 sch_htb
> sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm iptable_raw 
> xt_comment xt_recent xt_policy ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP 
> ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP
> ipt_ah ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp 
> nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp 
> nf_conntrack_amanda nf_conntrack_sane
> nf_conntrack_tftp nf_conntrack_sip nf_conntrack_proto_sctp nf_conntrack_pptp 
> nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns 
> nf_conntrack_irc nf_conntrack_h323
> nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss xt_pkttype xt_physdev 
> xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_MARK xt_mark 
> xt_mac xt_limit xt_length xt_iprange xt_helper
> xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack xt_CONNMARK xt_connmark 
> xt_CLASSIFY ipt_LOG xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 
> nf_defrag_ipv4 nf_conntrack iptable_mangle
> nfnetlink iptable_filter ip_tables x_tables snd_pcm snd_timer snd soundcore 
> snd_page_alloc pcspkr evdev ext4 mbcache jbd2 crc16 dm_mod raid1 md_mod 
> xen_netfront xen_blkfront [ 1957.432378] CPU
> 6: [ 1957.432378] Modules linked in: quota_v2 quota_tree ip6table_filter 
> ip6_tables act_police cls_flow cls_fw cls_u32 sch_htb sch_hfsc sch_ingress 
> sch_sfq xt_time xt_connlimit xt_realm
> iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REJECT ipt_REDIRECT 
> ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah ipt_addrtype 
> nf_nat_tftp nf_nat_snmp_basic nf_nat_sip
> nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda 
> ts_kmp nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp 
> nf_conntrack_sip nf_conntrack_proto_sctp
> nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink 
> nf_conntrack_netbios_ns nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp 
> xt_TPROXY nf_tproxy_core xt_tcpmss xt_pkttype xt_physdev
> xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_MARK xt_mark 
> xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp 
> xt_dccp xt_conntrack xt_CONNMARK xt_connmark
> xt_CLASSIFY ipt_LOG xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 
> nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables 
> x_tables snd_pcm snd_timer snd soundcore
> snd_page_alloc pcspkr evdev ext4 mbcache jbd2 crc16 dm_mod raid1 md_mod 
> xen_netfront xen_blkfront [ 1957.432378] Pid: 18704, comm: apache2 Tainted: G 
>      D    2.6.32-5-xen-amd64 #1 [
> 1957.432378] RIP: e030:[<ffffffff8130d116>]  [<ffffffff8130d116>] 
> _spin_lock+0x13/0x1b [ 1957.432378] RSP: e02b:ffff88018c249b10  EFLAGS: 
> 00000297 [ 1957.432378] RAX: 00000000000000d2 RBX:
> ffff88018c249b28 RCX: ffff88018c249b68 [ 1957.432378] RDX: 00000000000000d1 
> RSI: 0000000000000200 RDI: ffff8801b8c49580 [ 1957.432378] RBP: 
> 0000000000000200 R08: 0000000000000008 R09:
> ffffffff814eb900 [ 1957.432378] R10: ffff8801bf373880 R11: ffffffff811ba500 
> R12: ffff8801b8c49500 [ 1957.432378] R13: 000000000000c580 R14: 
> ffff88018c249b28 R15: ffffffff814eb8c0 [ 1957.432378]
> FS:  00007fe81d353740(0000) GS:ffff88000aa4c000(0000) knlGS:0000000000000000 
> [ 1957.432378] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b [ 
> 1957.432378] CR2: 00007fe820830038 CR3:
> 0000000001001000 CR4: 0000000000000660 [ 1957.432378] DR0: 0000000000000000 
> DR1: 0000000000000000 DR2: 0000000000000000 [ 1957.432378] DR3: 
> 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400 [ 1957.432378] Call Trace: [ 1957.432378]  
> [<ffffffff8100dd87>] ? xen_exit_mmap+0xf8/0x136 [ 1957.432378]  
> [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001 [ 1957.432378]
> [<ffffffff810d1464>] ? exit_mmap+0x5a/0x148 [ 1957.432378]  
> [<ffffffff8100ecf2>] ? check_events+0x12/0x20 [ 1957.432378]  
> [<ffffffff811ba500>] ? dummycon_dummy+0x0/0x3 [ 1957.432378]
> [<ffffffff8104cc7d>] ? mmput+0x3c/0xdf [ 1957.432378]  [<ffffffff81050882>] ? 
> exit_mm+0x102/0x10d [ 1957.432378]  [<ffffffff8100ec99>] ? 
> xen_irq_enable_direct_end+0x0/0x7 [ 1957.432378]
> [<ffffffff810522a7>] ? do_exit+0x1f8/0x6c6 [ 1957.432378]  
> [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1 [ 1957.432378]  
> [<ffffffff8130d13a>] ? _spin_unlock_irqrestore+0xd/0xe [
> 1957.432378]  [<ffffffff8104f523>] ? release_console_sem+0x17e/0x1af [ 
> 1957.432378]  [<ffffffff8130dfdd>] ? oops_end+0xaf/0xb4 [ 1957.432378]  
> [<ffffffff810135f0>] ? do_invalid_op+0x8b/0x95 [
> 1957.432378]  [<ffffffff8100c694>] ? pin_pagetable_pfn+0x2d/0x36 [ 
> 1957.432378]  [<ffffffff810bb157>] ? __alloc_pages_nodemask+0x11c/0x5f5 [ 
> 1957.432378]  [<ffffffff8101293b>] ?
> invalid_op+0x1b/0x20 [ 1957.432378]  [<ffffffff8100c694>] ? 
> pin_pagetable_pfn+0x2d/0x36 [ 1957.432378]  [<ffffffff8100c690>] ? 
> pin_pagetable_pfn+0x29/0x36 [ 1957.432378]  [<ffffffff810cd73e>] ?
> __pte_alloc+0x6b/0xc6 [ 1957.432378]  [<ffffffff810cb5f0>] ? 
> pmd_alloc+0x28/0x5b [ 1957.432378]  [<ffffffff810cd867>] ? 
> handle_mm_fault+0xce/0x80f [ 1957.432378]  [<ffffffff810d2e28>] ?
> do_brk+0x227/0x301 [ 1957.432378]  [<ffffffff8130f616>] ? 
> do_page_fault+0x2e0/0x2fc [ 1957.432378]  [<ffffffff8130d4b5>] ? 
> page_fault+0x25/0x30
> 
> 
> Any other possible workaround besides upgrading?
> 
> Thanks, regards, Sergi
> 
> _______________________________________________ Xen-users mailing list 
> Xen-users@xxxxxxxxxxxxx http://lists.xen.org/xen-users

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPYMhMAAoJEN00VGSIizZ0OcMIAKtUdri/WEF9CMjpuUN6qlXS
OeyNJHZU+5cPC5qIv0JLm+0wSoVS242tqMW2me83hOivTebtbZ9CwEHBI3+Zv841
X6rKa9N1wS75sj2q5Qfz3q10dcIWC2tsSl95x+IqeXGVmLCNlTdWWQglJYnLVQ4W
WqyLHcNT5LOypHnepaxr3OZNCdDCUz71vzh256vXoKUV6GpFkM8s4z13KvPtyl9C
ailGL7MN4wPjQtRoTn0isusmKLVRbKDNdEEX8rXgP4Du+aW5kRarvHAP6jL5E0jv
XSImZIJFXInMmhHSKLywotP/a0/8UJDsYthTl+A9vvX/X/VerPkDY+GaEvuHpC8=
=vSwo
-----END PGP SIGNATURE-----

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users