Re: [Xen-users] BUG: soft lockup - CPU#6 stuck for 61s! [apache2:18704]

[David Erickson <halcyon1981@xxxxxxxxx>]

FWIW I am seeing soft lockups under high VM<->VM network communication
as well, using XenServer 5.6FP1 with the hotfix applied that
supposedly fixes this.. the kernel version is:
2.6.32.12-0.7.1.xs5.6.100.316.170595xen

Not sure if this has been sufficiently resolved in later Xen versions or not.

On Wed, Mar 14, 2012 at 9:33 AM, Sergi Seira <s.seira@xxxxxxxxx> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hello,
>
> well, if someone runs into this bug, i've managed to control it by lowering 
> the memory fingerprint of some processes, like apache.
> It seems that the less commited memory the less soft lockups under heavy load.
>
> Regards,
> Sergi
>
> On 03/07/2012 07:23 PM, Sergi Seira wrote:
>> Hello,
>>
>> there has been some threads about this topic before, but no workaround so 
>> far has worked for me (clocksource jiffies, pit, ...)
>>
>> I'm running debian squeeze 2.6.32-5-xen-amd64.
>>
>> DomU usually have 6 or 7 cores and 6 or 7 GB of RAM.
>>
>> Only happens on servers with considerable load and memory usage.
>>
>> This is the the message that once starts showing degradation begins until 
>> load reaches impossible values and I have to destroy the domU because soft 
>> lockup makes processes uninterruptible :
>>
>> [ 1957.432378] BUG: soft lockup - CPU#6 stuck for 61s! [apache2:18704] [ 
>> 1957.432378] Modules linked in: quota_v2 quota_tree ip6table_filter 
>> ip6_tables act_police cls_flow cls_fw cls_u32 sch_htb
>> sch_hfsc sch_ingress sch_sfq xt_time xt_connlimit xt_realm iptable_raw 
>> xt_comment xt_recent xt_policy ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP 
>> ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP
>> ipt_ah ipt_addrtype nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp 
>> nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp 
>> nf_conntrack_amanda nf_conntrack_sane
>> nf_conntrack_tftp nf_conntrack_sip nf_conntrack_proto_sctp nf_conntrack_pptp 
>> nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns 
>> nf_conntrack_irc nf_conntrack_h323
>> nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss xt_pkttype xt_physdev 
>> xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_MARK xt_mark 
>> xt_mac xt_limit xt_length xt_iprange xt_helper
>> xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack xt_CONNMARK xt_connmark 
>> xt_CLASSIFY ipt_LOG xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 
>> nf_defrag_ipv4 nf_conntrack iptable_mangle
>> nfnetlink iptable_filter ip_tables x_tables snd_pcm snd_timer snd soundcore 
>> snd_page_alloc pcspkr evdev ext4 mbcache jbd2 crc16 dm_mod raid1 md_mod 
>> xen_netfront xen_blkfront [ 1957.432378] CPU
>> 6: [ 1957.432378] Modules linked in: quota_v2 quota_tree ip6table_filter 
>> ip6_tables act_police cls_flow cls_fw cls_u32 sch_htb sch_hfsc sch_ingress 
>> sch_sfq xt_time xt_connlimit xt_realm
>> iptable_raw xt_comment xt_recent xt_policy ipt_ULOG ipt_REJECT ipt_REDIRECT 
>> ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah ipt_addrtype 
>> nf_nat_tftp nf_nat_snmp_basic nf_nat_sip
>> nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda 
>> ts_kmp nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp 
>> nf_conntrack_sip nf_conntrack_proto_sctp
>> nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink 
>> nf_conntrack_netbios_ns nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp 
>> xt_TPROXY nf_tproxy_core xt_tcpmss xt_pkttype xt_physdev
>> xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_MARK xt_mark 
>> xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp 
>> xt_dccp xt_conntrack xt_CONNMARK xt_connmark
>> xt_CLASSIFY ipt_LOG xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 
>> nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink iptable_filter 
>> ip_tables x_tables snd_pcm snd_timer snd soundcore
>> snd_page_alloc pcspkr evdev ext4 mbcache jbd2 crc16 dm_mod raid1 md_mod 
>> xen_netfront xen_blkfront [ 1957.432378] Pid: 18704, comm: apache2 Tainted: 
>> G      D    2.6.32-5-xen-amd64 #1 [
>> 1957.432378] RIP: e030:[<ffffffff8130d116>]  [<ffffffff8130d116>] 
>> _spin_lock+0x13/0x1b [ 1957.432378] RSP: e02b:ffff88018c249b10  EFLAGS: 
>> 00000297 [ 1957.432378] RAX: 00000000000000d2 RBX:
>> ffff88018c249b28 RCX: ffff88018c249b68 [ 1957.432378] RDX: 00000000000000d1 
>> RSI: 0000000000000200 RDI: ffff8801b8c49580 [ 1957.432378] RBP: 
>> 0000000000000200 R08: 0000000000000008 R09:
>> ffffffff814eb900 [ 1957.432378] R10: ffff8801bf373880 R11: ffffffff811ba500 
>> R12: ffff8801b8c49500 [ 1957.432378] R13: 000000000000c580 R14: 
>> ffff88018c249b28 R15: ffffffff814eb8c0 [ 1957.432378]
>> FS:  00007fe81d353740(0000) GS:ffff88000aa4c000(0000) knlGS:0000000000000000 
>> [ 1957.432378] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b [ 
>> 1957.432378] CR2: 00007fe820830038 CR3:
>> 0000000001001000 CR4: 0000000000000660 [ 1957.432378] DR0: 0000000000000000 
>> DR1: 0000000000000000 DR2: 0000000000000000 [ 1957.432378] DR3: 
>> 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400 [ 1957.432378] Call Trace: [ 1957.432378] 
>>  [<ffffffff8100dd87>] ? xen_exit_mmap+0xf8/0x136 [ 1957.432378] 
>>  [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001 [ 1957.432378]
>> [<ffffffff810d1464>] ? exit_mmap+0x5a/0x148 [ 1957.432378] 
>>  [<ffffffff8100ecf2>] ? check_events+0x12/0x20 [ 1957.432378] 
>>  [<ffffffff811ba500>] ? dummycon_dummy+0x0/0x3 [ 1957.432378]
>> [<ffffffff8104cc7d>] ? mmput+0x3c/0xdf [ 1957.432378]  [<ffffffff81050882>] 
>> ? exit_mm+0x102/0x10d [ 1957.432378]  [<ffffffff8100ec99>] ? 
>> xen_irq_enable_direct_end+0x0/0x7 [ 1957.432378]
>> [<ffffffff810522a7>] ? do_exit+0x1f8/0x6c6 [ 1957.432378] 
>>  [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1 [ 1957.432378] 
>>  [<ffffffff8130d13a>] ? _spin_unlock_irqrestore+0xd/0xe [
>> 1957.432378]  [<ffffffff8104f523>] ? release_console_sem+0x17e/0x1af [ 
>> 1957.432378]  [<ffffffff8130dfdd>] ? oops_end+0xaf/0xb4 [ 1957.432378] 
>>  [<ffffffff810135f0>] ? do_invalid_op+0x8b/0x95 [
>> 1957.432378]  [<ffffffff8100c694>] ? pin_pagetable_pfn+0x2d/0x36 [ 
>> 1957.432378]  [<ffffffff810bb157>] ? __alloc_pages_nodemask+0x11c/0x5f5 [ 
>> 1957.432378]  [<ffffffff8101293b>] ?
>> invalid_op+0x1b/0x20 [ 1957.432378]  [<ffffffff8100c694>] ? 
>> pin_pagetable_pfn+0x2d/0x36 [ 1957.432378]  [<ffffffff8100c690>] ? 
>> pin_pagetable_pfn+0x29/0x36 [ 1957.432378]  [<ffffffff810cd73e>] ?
>> __pte_alloc+0x6b/0xc6 [ 1957.432378]  [<ffffffff810cb5f0>] ? 
>> pmd_alloc+0x28/0x5b [ 1957.432378]  [<ffffffff810cd867>] ? 
>> handle_mm_fault+0xce/0x80f [ 1957.432378]  [<ffffffff810d2e28>] ?
>> do_brk+0x227/0x301 [ 1957.432378]  [<ffffffff8130f616>] ? 
>> do_page_fault+0x2e0/0x2fc [ 1957.432378]  [<ffffffff8130d4b5>] ? 
>> page_fault+0x25/0x30
>>
>>
>> Any other possible workaround besides upgrading?
>>
>> Thanks, regards, Sergi
>>
>> _______________________________________________ Xen-users mailing list 
>> Xen-users@xxxxxxxxxxxxx http://lists.xen.org/xen-users
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBAgAGBQJPYMhMAAoJEN00VGSIizZ0OcMIAKtUdri/WEF9CMjpuUN6qlXS
> OeyNJHZU+5cPC5qIv0JLm+0wSoVS242tqMW2me83hOivTebtbZ9CwEHBI3+Zv841
> X6rKa9N1wS75sj2q5Qfz3q10dcIWC2tsSl95x+IqeXGVmLCNlTdWWQglJYnLVQ4W
> WqyLHcNT5LOypHnepaxr3OZNCdDCUz71vzh256vXoKUV6GpFkM8s4z13KvPtyl9C
> ailGL7MN4wPjQtRoTn0isusmKLVRbKDNdEEX8rXgP4Du+aW5kRarvHAP6jL5E0jv
> XSImZIJFXInMmhHSKLywotP/a0/8UJDsYthTl+A9vvX/X/VerPkDY+GaEvuHpC8=
> =vSwo
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxx
> http://lists.xen.org/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users