Re: [Xen-devel] Fwd: Debugging "Xen BUG at ... list.h" on ARM

[Thomas Leonard <tal36@xxxxxxxxx>]

On 14 April 2014 10:12, Julien Grall <julien.grall@xxxxxxxxxx> wrote:
> (Adding a couple of people in cc)
>
> On 14/04/14 09:02, Thomas Leonard wrote:
>>
>> Hi all,
>
>
> Hello,
>
>
>> I'm new to Xen, and trying to get the arm32 port of Mini-OS working on
>> a Cubieboard2. I have the board running an Ubuntu Dom0 and an Ubuntu
>> guest happily, and I've written up the instructions for that here:
>>
>>
>> https://github.com/talex5/mirage-www/blob/master/tmpl/wiki/xen-on-cubieboard2.md
>>
>> However, while trying to get a Mini-OS domU guest working I can now
>> reliably crash the hypervisor. No doubt this is a bug in my guest
>> code, but I assume it shouldn't be possible for a buggy guest to take
>> down Xen itself.
>>
>> Here's a simplified version of my code which shows the problem:
>>
>>    https://github.com/talex5/xen-paratest
>>
>> I'm building Xen from the stable-4.4 Git branch (03eb51340), compiled
>> with debug on.
>>
>> The code is rather simple: it's just trying to initialise the data
>> cache, as described in the "ARM Cortex-A Series Programmerâs Guide".
>> I'm not actually sure whether this is necessary for a paravirtualised
>> guest, but I was getting some odd problems which looked cache related,
>> so I added this code in the hope of fixing that.
>
>
> Does it crash if you remove your cache code?

No, it just produces the wrong output in that case.

For background: I was making a hypervisor call to write out a string,
then modifying a second string, then make a call to write that (both
using HYPERVISOR_console_io).

Xen printed the second string with its first part containing the old
text and the second part containing the new text. It looked like the
old part was the bytes on the same cache line as the first string I
printed, and I was trying to debug that.

>> The cfg file is just:
>>
>> kernel = "/root/paratest.img"
>> memory = 128
>> name = "paratest"
>> vcpus = 1
>> serial="pty"
>>
>> When the guest is started, Xen always crashes. Sometimes silently, but
>> usually writing a stack-trace to the console. I've included a
>> selection of them in the commit messages, but a typical one looks like
>> this:
>
> hmmm, I've just tried your kernel on midway and Xen didn't crash.
>
> Do you have more input? Number of physical CPUs used, state of the VCPU...?

There are 2 physical CPUs. I'm not sure what you mean by the VCPU
state (beyond what's in the dump).

>>      (XEN) Assertion 'svc ==
>> CSCHED_VCPU(curr_on_cpu(svc->vcpu->processor))' failed, line 859, file
>> sche
>>      (XEN) Xen BUG at sched_credit.c:859
>
>
> I can't find any assert at this line. Did you modify Xen code?

Sorry, you're right, I added an extra assert there. Here's a stack
trace with a clean build (stable-4.4 branch + debug=y):

(XEN) Assertion 'entry->next->prev == entry' failed, line 175, file
/home/tal/work/cubieboard/xen/xen/include/xen/list.h
(XEN) Xen BUG at /home/tal/work/cubieboard/xen/xen/include/xen/list.h:175
(XEN) CPU1: Unexpected Trap: Undefined Instruction
(XEN) ----[ Xen-4.4.1-pre  arm32  debug=y  Not tainted ]----
(XEN) CPU:    1
(XEN) PC:     002414cc __bug+0x28/0x44
(XEN) CPSR:   200e01da MODE:Hypervisor
(XEN)      R0: 002656dc R1: 00000001 R2: 3fd2bd80 R3: 200e01da
(XEN)      R4: 000000af R5: 0025d238 R6: 002e4020 R7: 002e6380
(XEN)      R8: 002e6380 R9: 600e01da R10:4003a940 R11:40037e34 R12:00000002
(XEN) HYP: SP: 40037e2c LR: 002414cc
(XEN)
(XEN)   VTCR_EL2: 80002558
(XEN)  VTTBR_EL2: 000100007ec06000
(XEN)
(XEN)  SCTLR_EL2: 30cd187f
(XEN)    HCR_EL2: 0000000000282835
(XEN)  TTBR0_EL2: 0000000076011000
(XEN)
(XEN)    ESR_EL2: 00000000
(XEN)  HPFAR_EL2: 000000000001c810
(XEN)      HDFAR: c8800f00
(XEN)      HIFAR: c87400a0
(XEN)
(XEN) Xen stack trace from sp=40037e2c:
(XEN)    00000000 40037e64 0022e32c 15882183 00000007 062e43e6 062e43e6 00000000
(XEN)    4003a000 40004000 00000002 002e62ac 00000000 40037e7c 00250480 077e6000
(XEN)    40004000 4003a000 4003b000 40037e94 00243e70 002e6500 40004000 4003b000
(XEN)    4003b000 40037ea4 002445b8 14f1129d 00000007 40037f0c 002266e8 00000000
(XEN)    4003b7c8 40037efc 002e62a8 ffffffff ffffffff 002e6588 0024f9d0 40037eec
(XEN)    0022884c 40004000 00000000 ffffffff ffffffff 40037f00 c07063f8 40037f54
(XEN)    00000000 002b1ff4 002ae000 002e7614 00265b80 002b1ff0 002e4254 40037f3c
(XEN)    00229b28 00000000 ffffffff 002e7614 002ae000 002e7614 c07063f8 c07414d7
(XEN)    c050078c c06fe000 c07414d7 40037f44 00229bc0 40037f54 0024ee00 c0706474
(XEN)    c06fe000 40037f58 00251150 ffffffed 077de000 c070746c 00000000 c0706474
(XEN)    c06fe000 c07063f8 c07414d7 c050078c c06fe000 c07414d7 c06fe000 c070d900
(XEN)    ffffffff 0000d43b c0019d28 600e0093 00000000 be9695f4 c0742080 c0012400
(XEN)    c06fffa8 c000f600 c074208c c0012760 c0742098 c00125e0 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 600e0013 800f0193 800f0093
(XEN)    600e0193 00000000 00000000 030c3080 451a8c04
(XEN) Xen call trace:
(XEN)    [<002414cc>] __bug+0x28/0x44 (PC)
(XEN)    [<002414cc>] __bug+0x28/0x44 (LR)
(XEN)    [<0022e32c>] set_timer+0x1d0/0x278
(XEN)    [<00250480>] virt_timer_save+0x88/0x90
(XEN)    [<00243e70>] schedule_tail+0x48/0x2f4
(XEN)    [<002445b8>] context_switch+0x114/0x118
(XEN)    [<002266e8>] schedule+0x8a0/0x8c8
(XEN)    [<00229b28>] __do_softirq+0xf4/0x100
(XEN)    [<00229bc0>] do_softirq+0x14/0x18
(XEN)    [<0024ee00>] leave_hypervisor_tail+0x50/0x6c
(XEN)    [<00251150>] return_to_guest+0xc/0xb8

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel