[Xen-devel] Nested virtualization instabilities

[Etzion Bar-Noy <etzion@xxxxxxxxxxxx>]

Hi.

I am using the unstable build-from-source of Xen community. It reports as 4.4, and was pulled from Git few days ago. The problem I will describe below appears on version 4.3.1, compiled from source as well. In both cases - I used spec file to create my own RPMs, because it's the right method of doing it.

Problem: When VM running under HVM which is a hypervisor (XenServer 6.2 or VMware ESXi5.5) attempts to start a (HVM) virtual machine, the following happens, 100% reproducible on this server.

XenServer: domain crash. XenServer restarts itself. It doesn't happen for PV on XenServer VM

VMware: An "unknown failure", which leads me to think that the nested virtualization properties are not forwarded correctly to DomU

Just to clarify:

Physical hardware: host

Nested hypervisor: level 1 virtualization: XenServer or VMware ESXi

Nested VM: level 2 virtualization: some generic VM inside XenServer or ESXi (level 1)

This is the configuration used for XS:
name = "XenServer"
uuid = "a0b8c668-afaf-887a-1f4b-856dac4c8401"
maxmem = 4096
memory = 4096
vcpus = 2
builder = "hvm"
boot = "c"
pae = 1
acpi = 1
apic = 1
hap = 0
viridian = 0
rtc_timeoffset = 0
localtime = 0
>>>sdl = 0
vnc = 1
vncunused = 1
keymap = "en-us"
disk = [ "tap2:aio:/dev/share/VMS/xenserver01.lun,hda,w", ",hdc:cdrom,r" ]
vif = [ "mac=00:16:3e:14:0d:72,bridge=lan,script=vif-bridge,model=e1000" ]
parallel = "none"
serial = "pty"
nestedhvm = 1

nx=1

# A note - this problem appears with and without the 'nx' flag.

A snippet from strace to qemu process:

poll([{fd=30, events=POLLIN|POLLERR|POLLHUP}, {fd=18, events=POLLIN|POLLERR|POLLHUP}, {fd=29, events=POLLIN|POLLERR|POLLHUP}, {fd=21,
events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN|POLLERR|POLLHUP}, {fd=5, events=
POLLIN|POLLERR|POLLHUP}, {fd=7, events=POLLIN}, {fd=11, events=POLLIN}, {fd=28, events=POLLIN}, {fd=6, events=POLLIN}], 10, 307) = ? ERESTART_RESTARTBLOCK (To be restarted)
--- SIGHUP (Hangup) @ 0 (0) ---
write(6, "\1\0\0\0\0\0\0\0", 8)         = 8
rt_sigreturn(0x2)                       = -1 EINTR (Interrupted system call)
gettimeofday({1386761621, 705266}, NULL) = 0
clock_gettime(CLOCK_MONOTONIC, {1840, 693236049}) = 0
clock_gettime(CLOCK_MONOTONIC, {1840, 693333096}) = 0
gettimeofday({1386761621, 705546}, NULL) = 0
write(2, "qemu: terminating on signal 1", 29) = 29
write(2, " from pid 6933\n", 15)        = 15

From the xl log for this domain:
Waiting for domain XenServer (domid 5) to die [pid 6933]
Domain 5 has shut down, reason code 3 0x3
Action for shutdown reason code 3 is restart
Domain 5 needs to be cleaned up: destroying the domain
Done. Rebooting now
Waiting for domain XenServer (domid 6) to die [pid 6933]

From 'xl dmesg'
(XEN) <vm_launch_fail> error code 7
(XEN) domain_crash_sync called from vmcs.c:1293
(XEN) Domain 8 (vcpu#0) crashed on cpu#4:
(XEN) ----[ Xen-4.4-unstable  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    4
(XEN) RIP:    0000:[<0000000000000000>]
(XEN) RFLAGS: 0000000000000002   CONTEXT: hvm guest
(XEN) rax: 0000000000000000   rbx: ffff830824f98000   rcx: ffff830824f9ff80
(XEN) rdx: ffff82d0801d0cf9   rsi: 0000000000000000   rdi: ffff82d0801de7fc
(XEN) rbp: ffff82d080105ab1   rsp: 0000000000000000   r8:  0000000000000004
(XEN) r9:  ffff82d080105b20   r10: ffff830824f9ff70   r11: 0000000000000000
(XEN) r12: ffff830824f9ff50   r13: ffff830654f9e000   r14: ffff830824f9ff30
(XEN) r15: ffff82d080187e43   cr0: 0000000000000039   cr4: 0000000000002050
(XEN) cr3: 00000000feffa000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(d9) HVM Loader
(d9) Detected Xen v4.4-unstable
(d9) Xenbus rings @0xfeffc000, event channel 4
(d9) System requested SeaBIOS
(d9) CPU speed is 3400 MHz
(d9) Relocating guest memory for lowmem MMIO space disabled
(XEN) irq.c:270: Dom9 PCI link 0 changed 0 -> 5
(d9) PCI-ISA link 0 routed to IRQ5
(XEN) irq.c:270: Dom9 PCI link 1 changed 0 -> 10
(d9) PCI-ISA link 1 routed to IRQ10
(XEN) irq.c:270: Dom9 PCI link 2 changed 0 -> 11
(d9) PCI-ISA link 2 routed to IRQ11
(XEN) irq.c:270: Dom9 PCI link 3 changed 0 -> 5
(d9) PCI-ISA link 3 routed to IRQ5
(d9) pci dev 01:3 INTA->IRQ10
(d9) pci dev 02:0 INTA->IRQ11
(d9) pci dev 04:0 INTA->IRQ5
(d9) RAM in high memory; setting high_mem resource base to 10f800000
(d9) pci dev 03:0 bar 10 size 002000000: 0f0000008
(d9) pci dev 02:0 bar 14 size 001000000: 0f2000008
(d9) pci dev 04:0 bar 30 size 000040000: 0f3000000
(d9) pci dev 04:0 bar 10 size 000020000: 0f3040000
(d9) pci dev 03:0 bar 30 size 000010000: 0f3060000
(d9) pci dev 03:0 bar 14 size 000001000: 0f3070000
(d9) pci dev 02:0 bar 10 size 000000100: 00000c001
(d9) pci dev 04:0 bar 14 size 000000040: 00000c101
(d9) pci dev 01:1 bar 20 size 000000010: 00000c141
(d9) Multiprocessor initialisation:
(d9)  - CPU0 ... 46-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done.
(d9)  - CPU1 ... 46-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done.
(d9) Testing HVM environment:
(d9)  - REP INSB across page boundaries ... passed
(d9)  - GS base MSRs and SWAPGS ... passed
(d9) Passed 2 of 2 tests
(d9) Writing SMBIOS tables ...
(d9) Loading SeaBIOS ...
(d9) Creating MP tables ...
(d9) Loading ACPI ...
(d9) vm86 TSS at fc00a080
(d9) BIOS map:
(d9)  10000-100d3: Scratch space
(d9)  e0000-fffff: Main BIOS
(d9) E820 table:
(d9)  [00]: 00000000:00000000 - 00000000:000a0000: RAM
(d9)  HOLE: 00000000:000a0000 - 00000000:000e0000
(d9)  [01]: 00000000:000e0000 - 00000000:00100000: RESERVED
(d9)  [02]: 00000000:00100000 - 00000000:f0000000: RAM
(d9)  HOLE: 00000000:f0000000 - 00000000:fc000000
(d9)  [03]: 00000000:fc000000 - 00000001:00000000: RESERVED
(d9)  [04]: 00000001:00000000 - 00000001:0f800000: RAM
(d9) Invoking SeaBIOS ...
(d9) SeaBIOS (version rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
(d9)
(d9) Found Xen hypervisor signature at 40000000
(d9) xen: copy e820...
(d9) Relocating init from 0x000e2e88 to 0xeffe1d80 (size 57784)
(d9) CPU Mhz=3400
(d9) Found 7 PCI devices (max PCI bus is 00)
(d9) Allocated Xen hypercall page at effff000
(d9) Detected Xen v4.4-unstable
(d9) xen: copy BIOS tables...
(d9) Copying SMBIOS entry point from 0x00010010 to 0x000f1020
(d9) Copying MPTABLE from 0xfc001170/fc001180 to 0x000f0f20
(d9) Copying PIR from 0x00010030 to 0x000f0ea0
(d9) Copying ACPI RSDP from 0x000100b0 to 0x000f0e70
(d9) Using pmtimer, ioport 0xb008, freq 3579 kHz
(d9) Scan for VGA option rom
(d9) WARNING! Found unaligned PCI rom (vd=1013:00b8)
(d9) Running option rom at c000:0003
(XEN) stdvga.c:147:d9 entering stdvga and caching modes
(d9) Turning on vga text mode console
(d9) SeaBIOS (version rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
(d9) Machine UUID a0b8c668-afaf-887a-1f4b-856dac4c8401
(d9) Found 0 lpt ports
(d9) Found 1 serial ports
(d9) ATA controller 1 at 1f0/3f4/c140 (irq 14 dev 9)
(d9) ATA controller 2 at 170/374/c148 (irq 15 dev 9)
(d9) ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (14336 MiBytes)
(d9) Searching bootorder for: /pci@i0cf8/*@1,1/drive@0/disk@0
(d9) DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD]
(d9) Searching bootorder for: /pci@i0cf8/*@1,1/drive@1/disk@0
(d9) PS2 keyboard initialized
(d9) All threads complete.
(d9) Scan for option roms
(d9) Running option rom at c900:0003
(d9) pmm call arg1=1
(d9) pmm call arg1=0
(d9) pmm call arg1=1
(d9) pmm call arg1=0
(d9) Searching bootorder for: /pci@i0cf8/*@4
(d9)
(d9) Press F12 for boot menu.
(d9)
(d9) Searching bootorder for: HALT
(d9) drive 0x000f0e20: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=29360128
(d9) Space available for UMB: ca000-ee800, f0000-f0dc0
(d9) Returned 61440 bytes of ZoneHigh
(d9) e820 map has 7 items:
(d9)   0: 0000000000000000 - 000000000009fc00 = 1 RAM
(d9)   1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
(d9)   2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
(d9)   3: 0000000000100000 - 00000000effff000 = 1 RAM
(d9)   4: 00000000effff000 - 00000000f0000000 = 2 RESERVED
(d9)   5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
(d9)   6: 0000000100000000 - 000000010f800000 = 1 RAM
(d9) enter handle_19:
(d9)   NULL
(d9) Booting from Hard Disk...
(d9) Booting from 0000:7c00
[root@monster ~]# xl dmesg
(XEN) <vm_launch_fail> error code 7
(XEN) domain_crash_sync called from vmcs.c:1293
(XEN) Domain 8 (vcpu#0) crashed on cpu#4:
(XEN) ----[ Xen-4.4-unstable  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    4
(XEN) RIP:    0000:[<0000000000000000>]
(XEN) RFLAGS: 0000000000000002   CONTEXT: hvm guest
(XEN) rax: 0000000000000000   rbx: ffff830824f98000   rcx: ffff830824f9ff80
(XEN) rdx: ffff82d0801d0cf9   rsi: 0000000000000000   rdi: ffff82d0801de7fc
(XEN) rbp: ffff82d080105ab1   rsp: 0000000000000000   r8:  0000000000000004
(XEN) r9:  ffff82d080105b20   r10: ffff830824f9ff70   r11: 0000000000000000
(XEN) r12: ffff830824f9ff50   r13: ffff830654f9e000   r14: ffff830824f9ff30
(XEN) r15: ffff82d080187e43   cr0: 0000000000000039   cr4: 0000000000002050
(XEN) cr3: 00000000feffa000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(d9) HVM Loader
(d9) Detected Xen v4.4-unstable
(d9) Xenbus rings @0xfeffc000, event channel 4
(d9) System requested SeaBIOS
(d9) CPU speed is 3400 MHz
(d9) Relocating guest memory for lowmem MMIO space disabled
(XEN) irq.c:270: Dom9 PCI link 0 changed 0 -> 5
(d9) PCI-ISA link 0 routed to IRQ5
(XEN) irq.c:270: Dom9 PCI link 1 changed 0 -> 10
(d9) PCI-ISA link 1 routed to IRQ10
(XEN) irq.c:270: Dom9 PCI link 2 changed 0 -> 11
(d9) PCI-ISA link 2 routed to IRQ11
(XEN) irq.c:270: Dom9 PCI link 3 changed 0 -> 5
(d9) PCI-ISA link 3 routed to IRQ5
(d9) pci dev 01:3 INTA->IRQ10
(d9) pci dev 02:0 INTA->IRQ11
(d9) pci dev 04:0 INTA->IRQ5
(d9) RAM in high memory; setting high_mem resource base to 10f800000
(d9) pci dev 03:0 bar 10 size 002000000: 0f0000008
(d9) pci dev 02:0 bar 14 size 001000000: 0f2000008
(d9) pci dev 04:0 bar 30 size 000040000: 0f3000000
(d9) pci dev 04:0 bar 10 size 000020000: 0f3040000
(d9) pci dev 03:0 bar 30 size 000010000: 0f3060000
(d9) pci dev 03:0 bar 14 size 000001000: 0f3070000
(d9) pci dev 02:0 bar 10 size 000000100: 00000c001
(d9) pci dev 04:0 bar 14 size 000000040: 00000c101
(d9) pci dev 01:1 bar 20 size 000000010: 00000c141
(d9) Multiprocessor initialisation:
(d9)  - CPU0 ... 46-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done.
(d9)  - CPU1 ... 46-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done.
(d9) Testing HVM environment:
(d9)  - REP INSB across page boundaries ... passed
(d9)  - GS base MSRs and SWAPGS ... passed
(d9) Passed 2 of 2 tests
(d9) Writing SMBIOS tables ...
(d9) Loading SeaBIOS ...
(d9) Creating MP tables ...
(d9) Loading ACPI ...
(d9) vm86 TSS at fc00a080
(d9) BIOS map:
(d9)  10000-100d3: Scratch space
(d9)  e0000-fffff: Main BIOS
(d9) E820 table:
(d9)  [00]: 00000000:00000000 - 00000000:000a0000: RAM
(d9)  HOLE: 00000000:000a0000 - 00000000:000e0000
(d9)  [01]: 00000000:000e0000 - 00000000:00100000: RESERVED
(d9)  [02]: 00000000:00100000 - 00000000:f0000000: RAM
(d9)  HOLE: 00000000:f0000000 - 00000000:fc000000
(d9)  [03]: 00000000:fc000000 - 00000001:00000000: RESERVED
(d9)  [04]: 00000001:00000000 - 00000001:0f800000: RAM
(d9) Invoking SeaBIOS ...
(d9) SeaBIOS (version rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
(d9)
(d9) Found Xen hypervisor signature at 40000000
(d9) xen: copy e820...
(d9) Relocating init from 0x000e2e88 to 0xeffe1d80 (size 57784)
(d9) CPU Mhz=3400
(d9) Found 7 PCI devices (max PCI bus is 00)
(d9) Allocated Xen hypercall page at effff000
(d9) Detected Xen v4.4-unstable
(d9) xen: copy BIOS tables...
(d9) Copying SMBIOS entry point from 0x00010010 to 0x000f1020
(d9) Copying MPTABLE from 0xfc001170/fc001180 to 0x000f0f20
(d9) Copying PIR from 0x00010030 to 0x000f0ea0
(d9) Copying ACPI RSDP from 0x000100b0 to 0x000f0e70
(d9) Using pmtimer, ioport 0xb008, freq 3579 kHz
(d9) Scan for VGA option rom
(d9) WARNING! Found unaligned PCI rom (vd=1013:00b8)
(d9) Running option rom at c000:0003
(XEN) stdvga.c:147:d9 entering stdvga and caching modes
(d9) Turning on vga text mode console
(d9) SeaBIOS (version rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
(d9) Machine UUID a0b8c668-afaf-887a-1f4b-856dac4c8401
(d9) Found 0 lpt ports
(d9) Found 1 serial ports
(d9) ATA controller 1 at 1f0/3f4/c140 (irq 14 dev 9)
(d9) ATA controller 2 at 170/374/c148 (irq 15 dev 9)
(d9) ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (14336 MiBytes)
(d9) Searching bootorder for: /pci@i0cf8/*@1,1/drive@0/disk@0
(d9) DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD]
(d9) Searching bootorder for: /pci@i0cf8/*@1,1/drive@1/disk@0
(d9) PS2 keyboard initialized
(d9) All threads complete.
(d9) Scan for option roms
(d9) Running option rom at c900:0003
(d9) pmm call arg1=1
(d9) pmm call arg1=0
(d9) pmm call arg1=1
(d9) pmm call arg1=0
(d9) Searching bootorder for: /pci@i0cf8/*@4
(d9)
(d9) Press F12 for boot menu.
(d9)
(d9) Searching bootorder for: HALT
(d9) drive 0x000f0e20: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=29360128
(d9) Space available for UMB: ca000-ee800, f0000-f0dc0
(d9) Returned 61440 bytes of ZoneHigh
(d9) e820 map has 7 items:
(d9)   0: 0000000000000000 - 000000000009fc00 = 1 RAM
(d9)   1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
(d9)   2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
(d9)   3: 0000000000100000 - 00000000effff000 = 1 RAM
(d9)   4: 00000000effff000 - 00000000f0000000 = 2 RESERVED
(d9)   5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
(d9)   6: 0000000100000000 - 000000010f800000 = 1 RAM
(d9) enter handle_19:
(d9)   NULL
(d9) Booting from Hard Disk...
(d9) Booting from 0000:7c00

System description:

1. Centos 6.4 x86_64, latest updates

2. Xen4centos packages - current kernel is 3.10.23-11.el6.centos.alt.x86_64

3. System with 32GB RAM

4. Virtual disks are ZFS volumes (aka - block devices). Same problem appeared when on local ext3 disks

5. Using OpenVSwitch version 2.0, compiled to RPM, again.

6. Intel CPU:
processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 62
model name    : Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz
stepping    : 4
microcode    : 0x416
cpu MHz        : 3400.158
cache size    : 12288 KB
physical id    : 0
siblings    : 12
core id        : 0
cpu cores    : 6
apicid        : 0
initial apicid    : 0
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl nonstop_tsc eagerfpu pni pclmulqdq monitor est ssse3 cx16 sse4_1 sse4_2 popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm ida arat epb xsaveopt pln pts dtherm fsgsbase erms
bogomips    : 6800.31
clflush size    : 64
cache_alignment    : 64
address sizes    : 46 bits physical, 48 bits virtual
power management:

It was suggested by Mr. Ian Campbell to CC both Mr. Jun Nakajima and Mr. Eddie Dong, so, my apologies if this action is improper.

Any tip or advice would be great. Any request for further info would be also great!

Thanks!

Etzion Bar-Noy

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel