Re: [Xen-devel] credit2 BUG_ON triggered

[George Dunlap <george.dunlap@xxxxxxxxxx>]

On Wed, 2011-04-20 at 17:12 +0100, John Weekes wrote:
> On 4/20/2011 2:36 AM, George Dunlap wrote:
> > Can you give the changset number / hash of your tip?  In the current
> > unstable tip, line 1606 is in the middle of a printk...
> 
> This is 4.1-testing. The version information was given in the "xl info" 
> output -- it's just the latest:
> 
> xen_major              : 4
> xen_minor              : 1
> xen_extra              : .1-rc1-pre
> xen_changeset          : Thu Apr 07 15:26:58 2011 +0100 23025:dbf2ddf652dc
> 
> Here's what the line corresponds to:
> 
>      BUG_ON(!is_idle_vcpu(scurr->vcpu) && scurr->rqd != rqd);
> 
> If you'd like me to repeat the test with xen-unstable, I can do that.

Yes, please, if you have the time.  xen-unstable has some debugging
output I put there after this failure was seen once in the xen.org
automated testing.  I've been over that code a number of times, and have
no idea how that bug could be triggering. :-)

Alternately, if you find unstable too unstable (or if you don't want to
find out), you can revert xen-4.1-testing c/s 22977:6af8e01d3e4a, which
turns off the debugging code I'd put in for the 4.1 release.

Peace,
 -George

> 
> -John
> 
> >   -George
> >
> > On Tue, Apr 19, 2011 at 9:58 PM, John Weekes
> > <lists.xen@xxxxxxxxxxxxxxxxxx>  wrote:
> >> I am testing credit2 on a dual Xeon L5640 machine. I have an HVM Debian
> >> Squeeze domU that reliably leads to a panic when it's run with the credit2
> >> scheduler, but not with credit.
> >>
> >> The reproduction steps on this machine are simple:
> >>
> >> 1. Fully boot up the machine.
> >> 2. Enter commands that cause dom0 to use 100% CPU. For instance:
> >>
> >> screen -AmdS burn1 perl -e 'while(1) {}'
> >> screen -AmdS burn2 perl -e 'while(1) {}'
> >> screen -AmdS burn3 perl -e 'while(1) {}'
> >> screen -AmdS burn4 perl -e 'while(1) {}'
> >>
> >> 3. Start up the prepared Squeeze domU (which is a stock install), with "xm"
> >> ("xl" doesn't work with debug=y because of a spurious assert, but has the
> >> same problem with debug=n):
> >>
> >> cd /servers/customers
> >> xm create testvds4.cfg
> >>
> >> The serial console then shows this:
> >>
> >> (XEN) irq.c:324: Dom1 callback via changed to Direct Vector 0xe9
> >> (XEN) Xen BUG at sched_credit2.c:1606
> >> (XEN) ----[ Xen-4.1.1-rc1-pre  x86_64  debug=y  Not tainted ]----
> >> (XEN) CPU:    12
> >> (XEN) RIP:    e008:[<ffff82c48011a383>] csched_schedule+0xdb/0xab1
> >> (XEN) RFLAGS: 0000000000010082   CONTEXT: hypervisor
> >> (XEN) rax: ffff830c2246c000   rbx: ffff830c2246bd10   rcx: 0000000000000000
> >> (XEN) rdx: 0000000000000001   rsi: ffff82c480241680   rdi: ffff8300bf74c000
> >> (XEN) rbp: ffff83043b28fe38   rsp: ffff83043b28fd58   r8:  0000000000000002
> >> (XEN) r9:  000000000000003e   r10: 0000000000000018   r11: 00000000000186a0
> >> (XEN) r12: 0000000000000000   r13: ffff83043ffe02d0   r14: 000000000000000c
> >> (XEN) r15: ffff83043ffe0010   cr0: 000000008005003b   cr4: 00000000000026f0
> >> (XEN) cr3: 0000000c22436000   cr2: 0000000000000000
> >> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
> >> (XEN) Xen stack trace from rsp=ffff83043b28fd58:
> >> (XEN)    ffff82c4801bc822 ffff83043b298040 0000000000000282 
> >> ffff83043b28fd88
> >> (XEN)    ffff82c48012248f ffff8300bf74c000 ffff83043b28fdb8 
> >> ffff82c4801b59bd
> >> (XEN)    00000014c77c5137 ffff83043b28fe68 ffff82c480241680 
> >> 0000000000000001
> >> (XEN)    00007cfbc4d70217 ffff82c48014b2c0 ffff83043b298060 
> >> ffff83043b28fde8
> >> (XEN)    ffff82c480124345 ffff83043b298060 ffff83043b28fe38 
> >> 0000000000000082
> >> (XEN)    00000000000186a0 0000000000000082 000000000000000c 
> >> ffff8300bf74c000
> >> (XEN)    0000000000000000 ffff82c480241680 ffff83043b298040 
> >> ffff83043b298060
> >> (XEN)    ffff83043b28feb8 ffff82c48012061c ffff83043b28feb8 
> >> 00000014c77c5137
> >> (XEN)    0000000000000293 ffff8300bf74d868 ffff82c48012248f 
> >> ffff8300bf74c000
> >> (XEN)    ffff83043b28fe98 ffff82c4801b19d0 ffff8300bf74c000 
> >> ffff82c4802a8e80
> >> (XEN)    00000000ffffffff ffff82c4802a8880 ffff83043b28ff18 
> >> ffffffffffffffff
> >> (XEN)    ffff83043b28fef8 ffff82c480121caf 0440080000000001 
> >> ffff8300bf74c000
> >> (XEN)    0000000000000046 ffff8800018501a0 ffffffff81311470 
> >> 0000000000000092
> >> (XEN)    ffff83043b28ff08 ffff82c480121d0c ffff88000184c600 
> >> ffff82c4801bb3f1
> >> (XEN)    0000000000000092 ffffffff81311470 ffff8800018501a0 
> >> 0000000000000046
> >> (XEN)    ffff88000184c600 0000000000000001 00000000000186a0 
> >> 0000000000000008
> >> (XEN)    0000000000000200 0000000000000008 0000000000000000 
> >> 0000000000000002
> >> (XEN)    0000000000000000 0000000000000002 0000000000000007 
> >> 0000beef0000beef
> >> (XEN)    ffffffff81009308 0000beef0000beef 0000000000000046 
> >> ffff880031e63e58
> >> (XEN)    000000000000beef 000000000000beef 000000000000beef 
> >> 000000000000beef
> >> (XEN) Xen call trace:
> >> (XEN)    [<ffff82c48011a383>] csched_schedule+0xdb/0xab1
> >> (XEN)    [<ffff82c48012061c>] schedule+0x122/0x60c
> >> (XEN)    [<ffff82c480121caf>] __do_softirq+0x8d/0x9e
> >> (XEN)    [<ffff82c480121d0c>] do_softirq+0x4c/0x4e
> >> (XEN)
> >> (XEN)
> >> (XEN) ****************************************
> >> (XEN) Panic on CPU 12:
> >> (XEN) Xen BUG at sched_credit2.c:1606
> >> (XEN) ****************************************
> >> (XEN)
> >> (XEN) Reboot in five seconds...
> >>
> >> Where do we go from here?
> >>
> >> -John
> >>
> >> For reference, xl info output:
> >>
> >> dallas-dodec226-5 ~ # xl info
> >> host                   : dallas-dodec226-5
> >> release                : 2.6.32.37-gbe57219
> >> version                : #1 SMP Tue Apr 19 00:14:46 CDT 2011
> >> machine                : x86_64
> >> nr_cpus                : 24
> >> nr_nodes               : 2
> >> cores_per_socket       : 6
> >> threads_per_core       : 2
> >> cpu_mhz                : 2266
> >> hw_caps                :
> >> bfebfbff:2c100800:00000000:00003f40:009ee3fd:00000000:00000001:00000000
> >> virt_caps              : hvm hvm_directio
> >> total_memory           : 49143
> >> free_memory            : 47106
> >> free_cpus              : 0
> >> xen_major              : 4
> >> xen_minor              : 1
> >> xen_extra              : .1-rc1-pre
> >> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> >> hvm-3.0-x86_32p hvm-3.0-x86_64
> >> xen_scheduler          : credit2
> >> xen_pagesize           : 4096
> >> platform_params        : virt_start=0xffff800000000000
> >> xen_changeset          : Thu Apr 07 15:26:58 2011 +0100 23025:dbf2ddf652dc
> >> xen_commandline        : dom0_mem=1500M dom0_max_vcpus=4
> >> iommu=dom0-passthrough sched=credit2 loglvl=all guest_loglvl=all
> >> com2=115200,8n1 console=com2
> >> cc_compiler            : gcc version 4.4.5 (Gentoo 4.4.5 p1.2, pie-0.4.5)
> >> cc_compile_by          : root
> >> cc_compile_domain      : nuclearfallout.net
> >> cc_compile_date        : Tue Apr 19 14:26:02 CDT 2011
> >> xend_config_format     : 4
> >>
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@xxxxxxxxxxxxxxxxxxx
> >> http://lists.xensource.com/xen-devel
> >>
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@xxxxxxxxxxxxxxxxxxx
> > http://lists.xensource.com/xen-devel
> 



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel