Re: [Xen-users] ARM: "xen_add_mach_to_phys_entry: cannot add ... already exists and panics"

[Denis Schneider <v1ne2go@xxxxxxxxx>]

Hi,
I am using a Cubieboard2 with Xen 4.4.
After building Xen, installing Debian etc. as described there:
http://openmirage.org/wiki/xen-on-cubieboard2, I tried to use Debian
and Gentoo as domU.
Xen and the dom0 kernel were compiled with the Linaro toolchain:
arm-linux-gnueabihf-gcc (crosstool-NG linaro-1.13.1-4.8-2013.10 -
Linaro GCC 2013.10) 4.8.2 20131014 (prerelease)
The kernel was compiled from
https://github.com/linux-sunxi/linux-sunxi.git, branch sunxi-devel at
a076583266.

Unfortunatly, the machine won't tolerate network load. If I do "nc -l
-p </dev/zero" on domU and "nc $domUIP 1234 > /dev/null" on dom0 or an
external host, I get a constant stream of:
[ 189.507495] xen_add_phys_to_mach_entry: cannot add pfn=0x0006930f ->
mfn=0x0004c3bc: pfn=0x00069310 -> mfn=0x0004c3bc already exists
[ 189.531185] xen_add_phys_to_mach_entry: cannot add pfn=0x0006921d ->
mfn=0x0004c489: pfn=0x0006928f -> mfn=0x0004c489 already exists
[ 189.533478] xen_add_mach_to_phys_entry: cannot add pfn=0x0006921a ->
mfn=0x0004c485: pfn=0x00069283 -> mfn=0x0004c485 already exists
[ 189.536304] xen_add_phys_to_mach_entry: cannot add pfn=0x00069212 ->
mfn=0x0004c485: pfn=0x00069213 -> mfn=0x0004c485 already exists
[ 189.536843] xen_add_mach_to_phys_entry: cannot add pfn=0x00069211 ->
mfn=0x0004c484: pfn=0x0006853a -> mfn=0x0004c484 already exists
[ 189.536909] xen_add_phys_to_mach_entry: cannot add pfn=0x00069208 ->
mfn=0x0004c391: pfn=0x00069210 -> mfn=0x0004c391 already exists
[ 189.539105] xen_add_phys_to_mach_entry: cannot add pfn=0x00069204 ->
mfn=0x0004c485: pfn=0x00069207 -> mfn=0x0004c485 already exists
[ 189.539164] xen_add_mach_to_phys_entry: cannot add pfn=0x0006853f ->
mfn=0x0004c390: pfn=0x0006920a -> mfn=0x0004c390 already exists
[ 189.539569] xen_add_mach_to_phys_entry: cannot add pfn=0x0006852f ->
mfn=0x0004c484: pfn=0x0006853e -> mfn=0x0004c484 already exists
[ 189.541451] xen_add_phys_to_mach_entry: cannot add pfn=0x00068538 ->
mfn=0x0004c484: pfn=0x0006852c -> mfn=0x0004c484 already exists
[ 190.382277] xen_add_mach_to_phys_entry: cannot add pfn=0x00069110 ->
mfn=0x0004c3cc: pfn=0x00069129 -> mfn=0x0004c3cc already exists
[ 190.437460] xen_add_mach_to_phys_entry: cannot add pfn=0x00069280 ->
mfn=0x0004c3ea: pfn=0x00068533 -> mfn=0x0004c3ea already exists
[ 191.022920] xen_add_phys_to_mach_entry: cannot add pfn=0x00069200 ->
mfn=0x0004c3d0: pfn=0x0006920f -> mfn=0x0004c3d0 already exists
[ 191.029744] xen_add_phys_to_mach_entry: cannot add pfn=0x00068532 ->
mfn=0x0004c38e: pfn=0x00068529 -> mfn=0x0004c38e already exists

followed by a panic when disk activity occurs:
[ 300.924382] Unable to handle kernel paging request at virtual address aadd5000
[ 300.924434] pgd = c0004000
[ 300.924450] [aadd5000] *pgd=00000000
[ 300.924474] Internal error: Oops: 2805 [#1] ARM
[ 300.924491] Modules linked in:
[ 300.924521] CPU: 0 PID: 0 Comm: swapper Not tainted 3.15.0-00201-ga076583 #17
[ 300.924546] task: c055d5e0 ti: c0552000 task.ti: c0552000
[ 300.924579] PC is at v7_dma_inv_range+0x30/0x48
[ 300.924601] LR is at __dma_page_dev_to_cpu+0x80/0x124
[ 300.924622] pc : [<c0017ef4>] lr : [<c00140e8>] psr: 40010193
[ 300.924622] sp : c0553d68 ip : c05b01fc fp : 00000000
[ 300.924673] r10: 00001000 r9 : 00000002 r8 : cbb35aa0
[ 300.924693] r7 : 00000000 r6 : 00001000 r5 : c05b01fc r4 : c055d300
[ 300.924714] r3 : 0000003f r2 : 00000040 r1 : aadd6000 r0 : aadd5000
[ 300.924738] Flags: nZcv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel
[ 300.924763] Control: 10c5387d Table: 6af78059 DAC: 00000015
[ 300.924783] Process swapper (pid: 0, stack limit = 0xc0552240)
[ 300.924804] Stack: (0xc0553d68 to 0xc0554000)
[ 300.924827] 3d60: c0017f84 c0553d84 c03bf560 4add5000 00000000 0004add5
[ 300.924855] 3d80: 4add5000 cb895e10 00000002 c01cdb30 00001000
00000002 00000000 000690da
[ 300.924883] 3da0: 00000482 00000000 00000600 c92af000 00000001
00000001 00000002 cb895e10
[ 300.924911] 3dc0: 00000000 cbb02190 0000249c c01cde34 00001000
00000002 00000000 20010193
[ 300.924939] 3de0: 00000001 cbb10a80 c92af000 cbb10000 00000001
000000b0 00000001 c023b2b4
[ 300.924967] 3e00: 00000000 20010193 c0553e24 c003b9a8 cb82e508
cbb10a80 cbb10000 cbb11640
[ 300.924995] 3e20: 00000001 c023b544 00000000 0000000f cbb10000
c023b90c cbb10000 cbb11640
[ 300.925023] 3e40: 00000000 cbb10000 00000008 d08a2100 00000001
cbb00010 cbae7c10 c024d8d4
[ 300.925051] 3e60: fff27870 ffffffff fff15a00 ffffffff 000361e4
00000000 ffeea214 00000000
[ 300.925079] 3e80: 00000001 00000001 cbb02190 d08a2000 c055a01c
cbb02190 0000249c c024e000
[ 300.925107] 3ea0: 00000000 3b9aca00 00000000 00000001 c055a0e0
cbb03040 c0565c14 00000000
[ 300.925135] 3ec0: 00000000 00000058 cb807c00 c0586790 00000001
c00496a0 00000046 20010193
[ 300.925163] 3ee0: c0561db8 cb807c00 c0565c14 00000000 d0804000
c057df88 00000000 c0552000
[ 300.925192] 3f00: ffffffed c00497cc cb807c00 c004bfe0 c004bf48
00000058 00000058 c0048ec4
[ 300.925220] 3f20: c0565a30 c000f048 d080400c c055a404 c0553f50
c0008540 c000f19c c000f1a0
[ 300.925249] 3f40: 60010013 ffffffff c0553f84 c0011e40 ffffffed
00000000 00000001 00000000
[ 300.925277] 3f60: c055a0b8 0000004c c0552000 00000000 c057df88
00000000 c0552000 ffffffed
[ 300.925305] 3f80: 01000000 c0553f98 c000f19c c000f1a0 60010013
ffffffff c055adcc c0040c5c
[ 300.925333] 3fa0: cbfff5c0 00000000 c058678e c0552000 00000000
c055a000 00000000 c0526a40
[ 300.925361] 3fc0: ffffffff ffffffff c05264ec 00000000 00000000
c0546bc8 00000000 10c5387d
[ 300.925389] 3fe0: c055a04c c0546bc4 c055e6c8 60004059 410fc074
60008070 00000000 00000000
[ 300.925431] [<c0017ef4>] (v7_dma_inv_range) from [<c00140e8>]
(__dma_page_dev_to_cpu+0x80/0x124)
[ 300.925471] [<c00140e8>] (__dma_page_dev_to_cpu) from [<c01cdb30>]
(xen_unmap_single+0xbc/0x108)
[ 300.925508] [<c01cdb30>] (xen_unmap_single) from [<c01cde34>]
(xen_swiotlb_unmap_sg_attrs+0x48/0x68)
[ 300.925544] [<c01cde34>] (xen_swiotlb_unmap_sg_attrs) from
[<c023b2b4>] (ata_sg_clean+0x8c/0x120)
[ 300.925580] [<c023b2b4>] (ata_sg_clean) from [<c023b544>]
(__ata_qc_complete+0x34/0x144)
[ 300.925613] [<c023b544>] (__ata_qc_complete) from [<c023b90c>]
(ata_qc_complete_multiple+0xa0/0xd4)
[ 300.925649] [<c023b90c>] (ata_qc_complete_multiple) from
[<c024d8d4>] (ahci_handle_port_interrupt+0x124/0x5b0)
[ 300.925688] [<c024d8d4>] (ahci_handle_port_interrupt) from
[<c024e000>] (ahci_interrupt+0xc0/0x158)
[ 300.925724] [<c024e000>] (ahci_interrupt) from [<c00496a0>]
(handle_irq_event_percpu+0x38/0x13c)
[ 300.925759] [<c00496a0>] (handle_irq_event_percpu) from [<c00497cc>]
(handle_irq_event+0x28/0x38)
[ 300.925794] [<c00497cc>] (handle_irq_event) from [<c004bfe0>]
(handle_fasteoi_irq+0x98/0x17c)
[ 300.925828] [<c004bfe0>] (handle_fasteoi_irq) from [<c0048ec4>]
(generic_handle_irq+0x2c/0x3c)
[ 300.925862] [<c0048ec4>] (generic_handle_irq) from [<c000f048>]
(handle_IRQ+0x38/0x84)
[ 300.925899] [<c000f048>] (handle_IRQ) from [<c0008540>]
(gic_handle_irq+0x2c/0x54)
[ 300.925929] [<c0008540>] (gic_handle_irq) from [<c0011e40>]
(__irq_svc+0x40/0x50)
[ 300.925953] Exception stack(0xc0553f50 to 0xc0553f98)
[ 300.925973] 3f40: ffffffed 00000000 00000001 00000000
[ 300.926001] 3f60: c055a0b8 0000004c c0552000 00000000 c057df88
00000000 c0552000 ffffffed
[ 300.926028] 3f80: 01000000 c0553f98 c000f19c c000f1a0 60010013 ffffffff
[ 300.926057] [<c0011e40>] (__irq_svc) from [<c000f1a0>]
(arch_cpu_idle+0x2c/0x30)
[ 300.926090] [<c000f1a0>] (arch_cpu_idle) from [<c0040c5c>]
(cpu_startup_entry+0xf8/0x1dc)
[ 300.926125] [<c0040c5c>] (cpu_startup_entry) from [<c0526a40>]
(start_kernel+0x32c/0x338)
[ 300.926155] Code: 1e070f3e e1110003 e1c11003 1e071f3e (ee070f36)
[ 300.926181] ---[ end trace edc3479207e4afcc ]---
[ 300.926200] Kernel panic - not syncing: Fatal exception in interrupt
[ 300.926222] ---[ end Kernel panic - not syncing: Fatal exception in interrupt

disk activity is generated with "find /usr > /dev/null" on domU.
As you see from the panic stack trace, the domU root device is a SATA
HDD connected to the AHCI port.
If you use an SD card, you get a similar panic, but with the mmc
interrupt handler.
A panic can also occur tens of seconds after the network IO seized if
heavy disk IO follows.

I tried compiling the dom0 kernel with Gentoo-gcc 4.7.3 and
Linaro-4.8.2 and Gentoo-4.8.3 without additional CFLAGS. The result is
the same.

To me, it looks as if xen-netback or something in this area corrupts
memory on large amounts of network IO.
Does anybody have a working setup on a Cubieboard2 or similar that is
really stable? Or a clue for me what is really happening here?
I'd happily give more information or try out more things.

I am not the only one with this problem and also tried different power
supplies. The dom0 kernel is stable if used without domU. So it
shouldn't be a hardware problem.

Kind regards,

Denis

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users