Re: [BUG] Nested Virtualization Bug on x86-64 AMD CPU

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 05/12/2023 1:51 pm, Reima ISHII wrote:
> Dear Xen Development Team,
>
> I am writing to report a bug that I have encountered in a Xen HVM
> guest with nested virtualization.
> Below is a detailed description of the bug, its potential impact, and
> the environment in which it was observed.
>
> [Bug Description]
> The issue emerges when operating an HVM guest with nested
> virtualization on an x86-64 AMD CPU, specifically in 64-bit mode (Long
> Mode). The sequence to reproduce the bug is as follows:
>
> 1. Enable nestedhvm on an HVM guest.
> 2. In the L1 guest hypervisor, set CR0.PE, PG to 1 in VMCB12 and
> execute  VMRUN, correctly resulting in a VM entry into the L2 guest.
> 3. In the L2 guest, perform a vmcall, which returns control back to
> the L1 hypervisor.
> 4. Subsequently, while still in 64-bit mode, the L1 hypervisor just
> changes the CR0.PG to 0 in VMCB12 and then executes VMRUN.

Thankyou for the bug report.

Who is still in 64-bit mode ?

It is legal for a 64-bit L1 to VMRUN into a 32-bit L2 with PG=0.

But I'm guessing that you mean L2 is also 64-bit, and we're clearing PG,
thus creating an illegal state (LMA=1 && PG=0) in VMCB12.

And yes, in that case (virtual) VMRUN at L1 ought to fail with
VMEXIT_INVALID.

>
> It is this specific action - executing VMRUN with CR0.PG set to 0 in
> Long Mode - that triggers the BUG() within the
> nsvm_vmcb_guest_intercepts_exitcode function in
> arch/x86/hvm/svm/nestedsvm.c.
> For an unknown reason, a vmexit occurs with the code 0x402, which is
> flagged as an illegal exitcode, thus triggering the BUG().
>
> [Potential Impact]
> This bug presents a vulnerability that could allow a DoS attack from
> the guest VM to the host hypervisor.
>
> [Environment Details]
> Here are the specs of the environment where the bug occurred:
> - Xen Version: Xen-4.18-unstable (commit
> 290f82375d828ef93f831a5ef028f1283aa1ea47)
> - Architecture: x86_64 (intel)
>
> [Error Log]
> (XEN) d1v0 Unexpected nested vmexit: reason 0x66
> (XEN) d1v0 Unexpected nested vmexit: reason 0x7b
> (XEN) d1v0 Unexpected nested vmexit: reason 0x7b
> (XEN) d1v0 Unexpected nested vmexit: reason 0x7b
> (XEN) d1v0 Unexpected nested vmexit: reason 0x7b
> (XEN) arch/x86/hvm/svm/nestedsvm.c:982:d1v0 Illegal exitcode 0x402
> (XEN) Xen BUG at arch/x86/hvm/svm/nestedsvm.c:983
> (XEN) Debugging connection not set up.
> (XEN) ----[ Xen-4.18-unstable  x86_64  debug=y gcov=y  Tainted:   C    ]----
> (XEN) CPU:    10
> (XEN) RIP:    e008:[<ffff82d0402997b8>]
> arch/x86/hvm/svm/nestedsvm.c#nsvm_vmcb_guest_intercepts_exitcode+0x29e/0x4c1
> (XEN) RFLAGS: 0000000000010202   CONTEXT: hypervisor (d1v0)
> (XEN) rax: ffff830839bdd040   rbx: ffff83084f593000   rcx: 0000000000000008
> (XEN) rdx: ffff830839bd7fff   rsi: ffff830839be5da8   rdi: ffff830839be5da0
> (XEN) rbp: ffff830839bd7e48   rsp: ffff830839bd7e30   r8:  0000000000000001
> (XEN) r9:  ffff830839be5da0   r10: 0000000000000001   r11: 0000000000000010
> (XEN) r12: ffff83084f593000   r13: ffff83084f583000   r14: ffff83084f593000
> (XEN) r15: 0000000000000001   cr0: 000000008005003b   cr4: 0000000000f506e0
> (XEN) cr3: 000000084f6d4000   cr2: 0000000000000000
> (XEN) fsb: 0000000000000000   gsb: ffff888490140000   gss: 0000000000000000
> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
> (XEN) Xen code around <ffff82d0402997b8>
> (arch/x86/hvm/svm/nestedsvm.c#nsvm_vmcb_guest_intercepts_exitcode+0x29e/0x4c1):
> (XEN)  48 83 05 e0 ee 3b 00 01 <0f> 0b 48 83 05 de ee 3b 00 01 48 83 05 96 ee 
> 3b
> (XEN) Xen stack trace from rsp=ffff830839bd7e30:
> (XEN)    0000000000000402 ffff83084f593000 ffff83084f583000 ffff830839bd7e70
> (XEN)    ffff82d04029b052 0000000000000402 ffff830839bd7ef8 ffff83084f583000
> (XEN)    ffff830839bd7ee8 ffff82d0402a1121 ffff82d0402a4afa ffff82d0402a4b00
> (XEN)    ffff82d0402a4afa ffff82c0002d8000 ffff82d0402a4afa ffff82d0402a4b00
> (XEN)    ffff82d0402a4afa ffff82d0402a4b00 ffff83084f593000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 00007cf7c64280e7
> (XEN)    ffff82d0402a4b4c 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 000000001f4604f7 0000000000000006 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000002040
> (XEN)    00000000000004f7 0000000000000000 0000000000000000 000000001f467473
> (XEN)    0000beef0000beef 000000000000ffff 000000bf0000beef 0000000000000082
> (XEN)    0000000000000c62 000000000000beef 000000000000beef 000000000000beef
> (XEN)    00000000ffffbeef 000000000000beef 0000e0100000000a ffff83084f593000
> (XEN)    00000037f95a2000 0000000000f506e0 0000000000000000 0000000000000000
> (XEN)    0000030300000000 0000000000000000
> (XEN) Xen call trace:
> (XEN)    [<ffff82d0402997b8>] R
> arch/x86/hvm/svm/nestedsvm.c#nsvm_vmcb_guest_intercepts_exitcode+0x29e/0x4c1
> (XEN)    [<ffff82d04029b052>] F nestedsvm_check_intercepts+0x29/0x214
> (XEN)    [<ffff82d0402a1121>] F svm_vmexit_handler+0x351/0x2502
> (XEN)    [<ffff82d0402a4b4c>] F svm_stgi_label+0x5/0x15
> (XEN)
> (XEN) debugtrace_dump() global buffer starting
> 1 cpupool_create(pool=0,sched=6)
> 2 Created cpupool 0 with scheduler SMP Credit Scheduler rev2 (credit2)
> 3 cpupool_add_domain(dom=0,pool=0) n_dom 1 rc 0
> 4-14 p2m: p2m_alloc_table(): allocating p2m table
> 15 cpupool_add_domain(dom=1,pool=0) n_dom 2 rc 0
> (XEN) wrap: 0
> (XEN) debugtrace_dump() global buffer finished
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 10:
> (XEN) Xen BUG at arch/x86/hvm/svm/nestedsvm.c:983
> (XEN) ****************************************
> (XEN)

As an incidental observation, that function is particularly absurd and
the two switches should be merged.

VMExit reason 0x402 is AVIC_NOACCEL and Xen has no support for AVIC in
the slightest right now.  i.e. Xen shouldn't have AVIC active in the
VMCB, and should never any AVIC related VMExits.

It is possible that we've got memory corruption, and have accidentally
activated AVIC in the VMCB.

But, is this by any chance all running nested under KVM in your fuzzer?

~Andrew