Re: [Xen-devel] Xen Nested Virtualization for HyperPlatform

[Kangjie Xi <imxikangjie@xxxxxxxxx>]

I tried another hypervisor
SimpleVisor(https://github.com/ionescu007/SimpleVisor), I compile it
without EPT, and load it in VMware VM, it works well, however in Xen
VM, the OS hangs again.

The serial console outputs:

...
(XEN) vmx.c:3956:d1v0 ### vmx_vmexit_handler()
exit_reason:EXIT_REASON_VMLAUNCH:20
(XEN) vvmx.c:1687:d1v0 nvmx_handle_vmlaunch
(XEN) vvmx.c:1163:d1v0 virtual_vmentry
(XEN) vvmx.c:1380:d1v0 virtual_vmexit
(XEN) vmx.c:3950:d1v0 ### vmx_vmexit_handler()
exit_reason:EXIT_REASON_VMWRITE:25
(XEN) vmx.c:3962:d1v0 ### vmx_vmexit_handler()
exit_reason:EXIT_REASON_VMRESUME:24
(XEN) vvmx.c:1163:d1v0 virtual_vmentry
(XEN) vvmx.c:1383:d1v0 virtual_vmexit
(XEN) vmx.c:3950:d1v0 ### vmx_vmexit_handler()
exit_reason:EXIT_REASON_VMWRITE:25
(XEN) vmx.c:3962:d1v0 ### vmx_vmexit_handler()
exit_reason:EXIT_REASON_VMRESUME:24
(XEN) vvmx.c:1163:d1v0 virtual_vmentry
(XEN) vmx.c:4044:d1v0 ### vmx_vmexit_handler()
exit_reason:EXIT_REASON_EPT_VIOLATION:48
(XEN) hvm.c:1664:d1v0 hvm_hap_nested_page_fault, gpa: fed000f0, gla:
ffffffffffd0d0f0
(XEN) vvmx.c:2344:d1v0 ### nvmx_n2_vmexit_handler EXIT_REASON_EPT_VIOLATION
(XEN) vmx.c:4044:d1v0 ### vmx_vmexit_handler()
exit_reason:EXIT_REASON_EPT_VIOLATION:48
(XEN) hvm.c:1664:d1v0 hvm_hap_nested_page_fault, gpa: fed000f0, gla:
ffffffffffd0d0f0
...

After several EPT VIOLATION, the serial console outputs nothing, the
VCPU seems freeze.

The debug-key 'd' dumps registers:

(XEN) *** Dumping CPU17 guest state (d1v0): ***
(XEN) ----[ Xen-4.10-unstable  x86_64  debug=y   Not tainted ]----
(XEN) CPU:    17
(XEN) RIP:    0010:[<fffff80202a1edb8>]
(XEN) RFLAGS: 0000000000200046   CONTEXT: hvm guest (d1v0)
(XEN) rax: 0000000000000000   rbx: 0000000000000210   rcx: 000000000000000f
(XEN) rdx: ffffaa81e6145eb8   rsi: ffffaa81e6145f08   rdi: fffff80202a1f510
(XEN) rbp: ffffaa81e6145eb0   rsp: ffffaa81e6145df0   r8:  ffffaa81e6145f08
(XEN) r9:  000000000000002f   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000008000   r13: 0000000000000000   r14: 0000000000000001
(XEN) r15: 0000000000000210   cr0: 0000000080050031   cr4: 00000000001526f8
(XEN) cr3: 0000000100aef000   cr2: 00000203ee877fdd
(XEN) ds: 002b   es: 002b   fs: 0053   gs: 002b   ss: 0018   cs: 0010

The address 0xfed000f0 is HPET memory address, so IO virtualization
error again? anyone can help?

Thanks


2017-11-09 21:12 GMT+08:00 Kangjie Xi <imxikangjie@xxxxxxxxx>:
> Hi,
>
> I am researching
> HyperPlatform(https://github.com/tandasat/HyperPlatform) recently, it
> is a thin hypervisor based on Intel VT-x and EPT technology, it can't
> create a new virtual machine but just turns the already running
> Windows OS into its guest VM.
>
> HyperPlatform works very well in VMware VM with Intel VT-x/EPT
> enabled, however can't run in Xen VM with nested virtualizaiton, after
> loading HyperPlatform, the VM hangs or BSOD.
>
> I debug it with serial console and printk, add the following printks
> to function nestedhvm_hap_nested_page_fault(), which will be called by
> EPT violation exit handler.
>
> ...
>   212     /* walk the L1 P2M table */
>   213     rv = nestedhap_walk_L1_p2m(v, *L2_gpa, &L1_gpa,
> &page_order_21, &p2ma_21,
>   214         access_r, access_w, access_x);
> + 215     gdprintk(XENLOG_ERR, "nestedhap_walk_L1_p2m, rv: %d, L2_gpa:
> %lx, L1_gpa: %lx\n", rv, *L2_gpa, L1_gpa);
> ...
>   230     /* ==> we have to walk L0 P2M */
>   231     rv = nestedhap_walk_L0_p2m(p2m, L1_gpa, &L0_gpa,
>   232         &p2mt_10, &p2ma_10, &page_order_10,
>   233         access_r, access_w, access_x);
> + 234     gdprintk(XENLOG_ERR, "nestedhap_walk_L0_p2m, rv: %d, L1_gpa:
> %lx, L0_gpa: %lx\n", rv, L1_gpa, L0_gpa);
> ...
>
> The serial console output:
>
> ...
> (XEN) nested_hap.c:215:d1v0 nestedhap_walk_L1_p2m, rv: 0, L2_gpa:
> f1000000, L1_gpa: f1000000
> (XEN) nested_hap.c:234:d1v0 nestedhap_walk_L0_p2m, rv: 3, L1_gpa:
> f1000000, L0_gpa: 0
> ...
> (XEN) nested_hap.c:215:d1v0 nestedhap_walk_L1_p2m, rv: 0, L2_gpa:
> f1001000, L1_gpa: f1001000
> (XEN) nested_hap.c:234:d1v0 nestedhap_walk_L0_p2m, rv: 3, L1_gpa:
> f1001000, L0_gpa: 0
> ...
>
> There are many "rv: 3", which is NESTEDHVM_PAGEFAULT_L0_ERROR,
> returned by read-only check conditon:
>
> 170    if ( access_w && p2m_is_readonly(*p2mt) )
> 171       goto out;
>
> The address 0xf100000 is the VM's display memory address
> (0xF1000000-0xF1FFFFFF).
>
> Is it a I/O virtualization error or MMIO error? Any idea to fix it?
>
>
> Thanks
>
> -Xi Kangjie

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel