[Xen-devel] Xen Crash at the boot time

[domain@xxxxxxxx]

Topic was in Xen-users@xxxxxxxxxxxxx mailing list 
(https://lists.xen.org/archives/html/xen-users/2017-03/msg00049.html) 
but by advice was moved here.

We are trying to run Xen on our two servers Supermicro X10QBi, Version 
3.0, Build Date 05/04/2016 12:38:52, CPUs 4x Intel (R) Xeon (R) CPU E7 
-8890 v4 @ 2.20GHz. The first servers has RAM 6 TB (DDR4
1600MHz), the second has 3T (DDR4 1333MHz). Xen crashes at the boot 
time.
Part of XEN logs:
...
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch 
input to Xen)
(XEN) Freed 312kB init memory
mapping kernel into physical memory
about to get started...
(XEN) d0v0 Unhandled invalid opcode fault/trap [#6, ec=0000]
(XEN) domain_crash_sync called from entry.S: fault at ffff82d0802363aa 
entry.o#create_bounce_frame+0x137/0x146
(XEN) Domain 0 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-4.8.0  x86_64  debug=n   Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e033:[<ffffffff81f65e33>]
(XEN) RFLAGS: 0000000000000246   EM: 1   CONTEXT: pv guest (d0v0)
(XEN) rax: 0000000000000081 rbx: 0000000000000000 rcx: 00003fffc0000000 
(XEN) rdx: 0010000000000001 rsi: 0000000000000080 rdi: 00100403400000e7 
(XEN) rbp: ffffffff81e03dd8 rsp: ffffffff81e03cf8 r8:
0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 
0000000000000000 (XEN) r12: 0000000000000000 r13: 00003ffffffff000 r14: 
0000000080000000 (XEN) r15: 0000000100001000 cr0:
0000000080050033 cr4: 00000000003526e0
(XEN) cr3: 0000040339e0a000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff81e03cf8:
(XEN) 00003fffc0000000 0000000000000000 ffffffff81f65e33 
000000010000e030 (XEN) 0000000000010046 ffffffff81e03d38 
000000000000e02b ffffffff81f65d60 (XEN) 000001ff8799d000 
0000000100000000 0000000080000000
ffff880001e0a000 (XEN) 0000020000000001 ffff880001e0a018 
0000000000080403 0000000000000001 (XEN) ffff880000000000 
ffffffffff240000 ffffffffff200000 0000000200000000 (XEN) 
00000000fffffe02 0000000000180403
0000000100000000 0000000010000000 (XEN) 0000000004cdb000 
0000000080001000 0000060080000000 0000000000000018 (XEN) 
ffffffff81e03e70 ffffffff81f65171 ffffffff81e03e48 ffffffff81e03e48 
(XEN) ffffffff810233df
0000000000000000 0000000000000001 000000001008799d (XEN) 
0000000010000000 0000010000000000 7ff0000000000002 ffffffff00000018 
(XEN) ffffffff8200b020 2f3362bd3faaa8fa 0000000001000000 
ffffffff81e03f00 (XEN)
ffffffff82005920 0000000000000000 0000000000000000 ffffffff81e03e88 
(XEN) ffffffff81f6ba6b 0000000001000000 ffffffff81e03ef0 
ffffffff81f692ae (XEN) ffffffff00000010 ffffffff81e03f00 
ffffffff81e03eb8
2f3362bd3faaa8fa (XEN) 697a696c61697469 2f3362bd3faaa8fa 
ffffffffffffffff 0000000000000000 (XEN) ffffffff82005920 
0000000000000000 0000000000000000 ffffffff81e03f30 (XEN) 
ffffffff81f5dc74 2f3362bd3faaa8fa
0000000000000000 ffffffff81e03f60 (XEN) ffffffff81e03f5c 
0000000000000000 0000000000000000 ffffffff81e03f40 (XEN) 
ffffffff81f5d339 ffffffff81e03ff8 ffffffff81f64156 000406f100000000 
(XEN) 0000000100400800
0300000100000032 0000000000000005 0000000000000020
(XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds.
(XEN) Resetting with ACPI MEMORY or I/O RESET_REG.

root@data1:/usr/lib/debug/boot# addr2line -f -e vmlinux-4.4.0-67-generic 
ffffffff81f65e33
xen_relocate_p2m
/build/linux-dKXp7o/linux-4.4.0/arch/x86/xen/mmu.c:2175 (discriminator 
1)
ffffffff81e03d38:
__end_rodata_hpage_align
??:?

ffffffff81f65d60:
xen_early_virt_to_phys
/build/linux-dKXp7o/linux-4.4.0/arch/x86/xen/mmu.c:2063

ffffffff81e03e70:
__end_rodata_hpage_align
??:?

ffffffff81f65171:
xen_memory_setup
/build/linux-dKXp7o/linux-4.4.0/arch/x86/xen/setup.c:886

ffffffff81e03e48:
__end_rodata_hpage_align
??:?

ffffffff810233df:
__raw_callee_save_xen_restore_fl
??:?

ffffffff8200b020:
early_dynamic_pgts
??:?

ffffffff81e03f00:
__end_rodata_hpage_align
??:?

ffffffff82005920:
early_dynamic_pgts
??:?

 ffffffff81e03e88:
__end_rodata_hpage_align
??:?

 ffffffff81f6ba6b:
setup_memory_map
/build/linux-dKXp7o/linux-4.4.0/arch/x86/kernel/e820.c:1086

ffffffff81e03ef0:
__end_rodata_hpage_align
??:?

ffffffff81f692ae:
parse_setup_data
/build/linux-dKXp7o/linux-4.4.0/arch/x86/kernel/setup.c:414

ffffffff81e03f00:
__end_rodata_hpage_align
??:?

ffffffff81e03eb8:
__end_rodata_hpage_align
??:?

ffffffff82005920:
early_dynamic_pgts
??:?

ffffffff81e03f30:
__end_rodata_hpage_align
??:?

ffffffff81f5dc74:
bitmap_zero
/build/linux-dKXp7o/linux-4.4.0/include/linux/bitmap.h:197

ffffffff81e03f60:
__end_rodata_hpage_align
??:?

ffffffff81e03f5c:
__end_rodata_hpage_align
??:?

ffffffff81e03f40:
__end_rodata_hpage_align
??:?

ffffffff81f5d339:
x86_64_start_reservations
/build/linux-dKXp7o/linux-4.4.0/arch/x86/kernel/head64.c:196

ffffffff81e03ff8:
__end_rodata_hpage_align
??:?

ffffffff81f64156:
xen_start_kernel
/build/linux-dKXp7o/linux-4.4.0/arch/x86/xen/enlighten.c:1725 
(discriminator 1)
addr2line -f -e vmlinux ffffffff81f8ed2a
xen_relocate_p2m
mmu.c:2165 (discriminator 1)

We took some further investigations and found next code in the source 
file mmu.c
        /* Release the old p2m list and set new list info. */
p2m_pfn = PFN_DOWN(xen_early_virt_to_phys(xen_start_info->mfn_list));
       BUG_ON(!p2m_pfn);

Does someone know why value of variable p2m_pfn is 0?

Is that the bug of Xen part of kernel or something else?

The problem Exists on Ubuntu 16.04:

Xen     Kernel
------------------------
4.6.5   4.8.0-32-generic
4.6.5   4.4.49-ltmd-1
4.8     4.8.17
4.8     4.8.17
4.8     4.10.7
4.8     4.11.0-rc4

But with Xen xen 4.6.0-1ubuntu4.3 (Jan 12 06:59) and kernel 4.4.0-67.88 
the problem disappears. It seems that the newest Xen versions have that 
problem.
The problem Exists also on Gentoo:
xen 4.7.1
kernel 4.9.16
kernel 4.4.52

The Kernel configuration and Xen boot log attached.

Attachment: 2017.03.29-data1-1.txt
Description: Text document

Attachment: config-4.4.0-67-generic.txt
Description: Text document

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel