Re: [Xen-devel] Nested Virtualization Support Host-Only Hypervisors

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 09/06/17 08:44, Kangjie Xi wrote:
> Xen supports Nested
> Virtualization(https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen),
> which means I can run Xen or KVM hypervisor inside a virtual machine,
> and then create a L2 guest. However when I run a Host-Only Hypervisor
> (no L2 guests, like MoRE, SimpleVisor and HyperPlatform) inside of L1
> guest, the L1 hangs.
>
> Is it able to make Xen Nested Virtualization to support Host-Only
> hypervisors? It is useful for security research.
>
> * MoRE, https://github.com/ainfosec/MoRE
> * SimpleVisor, https://github.com/ionescu007/SimpleVisor
> * HyperPlatform, https://github.com/tandasat/HyperPlatform

Nested virt in Xen is very experimental, and almost doesn't function in
practice.

* No 32bit L1 hypervisor software will work.  Several VT-x instruction
use the operand size when reading parameters in memory, rather than the
proper 8, meaning that operations get attempted with an uninitialised
upper half.

* Running anything Linux-based requires you to compile L1 without
CONFIG_XEN.  The issue is that Linux ends up using event channels for
IPIs, and these don't get delivered when the vcpu is running L2.  If L2
spins in a loop waiting for something to happen, this ends up not
exiting to L1 until an NMI occurs.

* HyperV chokes on APIC related things.  I think the root cause is the
complete fiction of topology information currently exposed to guests,
but in the end, it either takes a #GP for trying to use x2apic MSRs
after disabling x2apic mode, or complains of a spurious #PF when trying
to access the xapic MMIO page.

* ESXi manages to take out L0 quite reliably.  Disabling VMCS shadowing
helps, but L1 still locks up when it tries booting L2.

There are at least 6 outstanding ways I am aware of for L1 to crash L0,
because of insufficient or incorrect parameter auditing.


I do not recommend anyone uses nested virt while it is in its current
state, other than to actively develop it.  There is active work ongoing
from XenServer to get nested virt into a usable state, but my estimation
of multiple person-years of effort still applies.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel