Re: [Xen-devel] [PATCH RFC] x86/emulate: implement hvmemul_cmpxchg() with an actual CMPXCHG

[Razvan Cojocaru <rcojocaru@xxxxxxxxxxxxxxx>]

On 04/03/2017 09:36 PM, Razvan Cojocaru wrote:
> On 04/03/2017 09:20 PM, Razvan Cojocaru wrote:
>> On 04/01/2017 07:56 PM, Razvan Cojocaru wrote:
>>> On 03/31/2017 06:04 PM, Jan Beulich wrote:
>>>>>>> On 31.03.17 at 17:01, <rcojocaru@xxxxxxxxxxxxxxx> wrote:
>>>>> On 03/31/2017 05:46 PM, Jan Beulich wrote:
>>>>>>>>> On 31.03.17 at 11:56, <rcojocaru@xxxxxxxxxxxxxxx> wrote:
>>>>>>> On 03/31/2017 10:34 AM, Jan Beulich wrote:
>>>>>>>>>>> On 31.03.17 at 08:17, <rcojocaru@xxxxxxxxxxxxxxx> wrote:
>>>>>>>>> On 03/30/2017 06:47 PM, Jan Beulich wrote:
>>>>>>>>>>> Speaking of emulated MMIO, I've got this when the guest was crashing
>>>>>>>>>>> immediately (pre RETRY loop):
>>>>>>>>>>>
>>>>>>>>>>>  MMIO emulation failed: d3v8 32bit @ 0008:82679f3c -> f0 0f ba 30 
>>>>>>>>>>> 00 72
>>>>>>>>>>> 07 8b cb e8 da 4b ff ff 8b 45
>>>>>>>>>>
>>>>>>>>>> That's a BTR, which we should be emulating fine. More information
>>>>>>>>>> would need to be collected to have a chance to understand what
>>>>>>>>>> might be going one (first of all the virtual and physical memory
>>>>>>>>>> address this was trying to act on).
>>>>>>>>>
>>>>>>>>> Right, the BTR part should be fine, but I think the LOCK part is 
>>>>>>>>> what's
>>>>>>>>> causing the issue. I've done a few more test runs to see what return
>>>>>>>>> RETRY (dumping the instruction with an "(r)" prefix to distinguish 
>>>>>>>>> from
>>>>>>>>> the UNHANDLEABLE dump), and a couple of instructions return RETRY (BTR
>>>>>>>>> and XADD, both LOCK-prefixed, which means they now involve CMPXCHG
>>>>>>>>> handler, which presumably now fails - possibly simply because it's
>>>>>>>>> always LOCKed in my patch):
>>>>>>>>
>>>>>>>> Well, all of that looks to be expected behavior. I'm afraid I don't see
>>>>>>>> how this information helps understanding the MMIO emulation failure
>>>>>>>> above.
>>>>>>>
>>>>>>> I've managed to obtain this log of emulation errors:
>>>>>>> https://pastebin.com/Esy1SkHx 
>>>>>>>
>>>>>>> The "virtual address" lines that are not followed by any "Mem event"
>>>>>>> line correspond to CMXCHG_FAILED return codes.
>>>>>>>
>>>>>>> The very last line is a MMIO emulation failed.
>>>>>>>
>>>>>>> It's probably important that this happens with the model where
>>>>>>> hvm_emulate_one_vm_event() does _not_ re-try the emulation until it
>>>>>>> succeeds. The other model allows me to go further with the guest, but
>>>>>>> eventually I get timeout-related BSODs or the guest becomes 
>>>>>>> unresponsive.
>>>>>>
>>>>>> Interesting. You didn't clarify what the printed "offset" values are,
>>>>>> and it doesn't look like these have any correlation with the underlying
>>>>>> (guest) physical address, which we would also want to see. And then
>>>>>> it strikes me as odd that in these last lines
>>>>>>
>>>>>> (XEN) Mem event (RETRY) emulation failed: d5v8 32bit @ 0008:826bb861 -> 
>>>>>> f0 0f 
>>>>> ba 30 00 72 07 8b cb e8 da 4b ff ff 8b 45
>>>>>> (XEN) virtual address: 0xffd080f0, offset: 4291854576
>>>>>> (XEN) MMIO emulation failed: d5v8 32bit @ 0008:82655f3c -> f0 0f ba 30 
>>>>>> 00 72 
>>>>> 07 8b cb e8 da 4b ff ff 8b 45
>>>>>>
>>>>>> the instruction pointers and virtual addresses are different, but the
>>>>>> code bytes are exactly the same. This doesn't seem very likely, so I
>>>>>> wonder whether there's an issue with us wrongly re-using previously
>>>>>> fetched insn bytes. (Of course I'd be happy to be proven wrong with
>>>>>> this guessing, by you checking the involved binary/ies.)
>>>>>
>>>>> Offset is the actual value of the "offset" parameter of
>>>>> hvmemul_cmpxchg().
>>>>
>>>> That's not very useful then, as for flat segments "offset" ==
>>>> "virtual address" (i.e. you merely re-print in decimal what you've
>>>> already printed in hex).
>>>
>>> The attached patch (a combination of your patch and mine) produces the
>>> following output when booting a Windows 7 32-bit guest with monitoring:
>>> https://pastebin.com/ayiFmj1N
>>>
>>> The failed MMIO emulation is caused by a mapping failure due to the
>>> "!nestedhvm_vcpu_in_guestmode(curr) && hvm_mmio_internal(gpa)" condition
>>> being true in hvmemul_vaddr_to_mfn(). I've ripped that off from
>>> __hvm_copy() but it looks like that might not be the right way to use it.
>>
>> Sorry to reply to this email instead of your original reply but I've
>> "left it" in my computer at work. Here's the last part of the log, with
>> the VCPU number logged for the GFN as well:
>>
>> (XEN) [8] gfn: 0x2781
>> (XEN) [8] virtual address: 0x827810a8, rc: 0
>> (XEN) [8] gfn: 0x2781
>> (XEN) [8] virtual address: 0x827810a8, rc: 0
>> (XEN) [8] gfn: 0x2781
>> (XEN) [8] virtual address: 0x827810cc, rc: 0
>> (XEN) [8] gfn: 0x2781
>> (XEN) [8] virtual address: 0x8278109c, rc: 0
>> (XEN) [8] gfn: 0x2781
>> (XEN) [8] virtual address: 0x827810d0, rc: 0
>> (XEN) [11] gfn: 0x2781
>> (XEN) [8] gfn: 0x2781
>> (XEN) [11] virtual address: 0x8278109c, rc: 0
>> (XEN) [8] virtual address: 0x8278109c, rc: 4
>> (XEN) Dump follows for VCPU 8
>> (XEN) Mem event (RETRY) emulation failed: d3v8 32bit @ 0008:826b5c7c ->
>> f0 0f c1 08 85 c9 74 1f f6 c1 02 75 1a 41 8d 41
>> (XEN) [11] gfn: 0x2781
>> (XEN) [11] virtual address: 0x827810a8, rc: 0
>> (XEN) [11] gfn: 0x2781
>> (XEN) [11] virtual address: 0x827810a8, rc: 0
>> (XEN) [8] gfn: 0xfed00
>> (XEN) !page
>> (XEN) hvmemul_vaddr_to_mfn() fail
>> (XEN) [8] virtual address: 0xffd080f0, rc: 1
>> (XEN) MMIO emulation failed: d3v8 32bit @ 0008:8264ff3c -> f0 0f c1 08
>> 85 c9 74 1f f6 c1 02 75 1a 41 8d 41
>>
>> The code does look the same for the last two failures on VCPU 8, for
>> clearly different GFNs and virtual addresses. The first time it hits a
>> protected page, we try to emulate the instruction and it fails with
>> X86EMUL_CMPXCHG_FAILED (rc: 4). Then it somehow pops up again and this
>> time it's MMIO-emulated, and that fails as well (with UNHANDLEABLE,
>> since we can't seem to be able to map the memory).
> 
> Another log, making sure that the MMIO emulation fail really happens on
> the same processor as the preceding (identical bytes) fail:
> 
> (XEN) [8] gfn: 0x276f
> (XEN) [11] virtual address: 0x8276f09c, rc: 0
> (XEN) [8] virtual address: 0x8276f09c, rc: 4
> (XEN) Dump follows for VCPU 8
> (XEN) Mem event (RETRY) emulation failed: d3v8 32bit @ 0008:826a3861 ->
> f0 0f ba 30 00 72 07 8b cb e8 da 4b ff ff 8b 45
> (XEN) [11] gfn: 0x276f
> (XEN) [11] virtual address: 0x8276f0d0, rc: 0
> (XEN) [11] gfn: 0x276f
> (XEN) [11] virtual address: 0x8276f09c, rc: 0
> (XEN) [11] gfn: 0x276f
> (XEN) [11] virtual address: 0x8276f0cc, rc: 0
> (XEN) [11] gfn: 0x276f
> (XEN) [11] virtual address: 0x8276f09c, rc: 0
> (XEN) [8] gfn: 0xfed00
> (XEN) !page
> (XEN) hvmemul_vaddr_to_mfn() fail
> (XEN) [8] virtual address: 0xffd080f0, rc: 1
> (XEN) Dump follows for VCPU 8
> (XEN) MMIO emulation failed: d3v8 32bit @ 0008:8263df3c -> f0 0f ba 30
> 00 72 07 8b cb e8 da 4b ff ff 8b 45
> (XEN) [11] gfn: 0x276f
> (XEN) [11] virtual address: 0x8276f0d0, rc: 0

As you've suggested, this does indeed seem to happen because of the
current model of emulating because of vm_event replies: an instruction
needs to be emulated in hvm_do_resume(), but inside
hvm_emulate_one_vm_event(), hvm_emulate_one() returns RETRY, presumably
because of the new cmpxchg handler. This causes
hvm_emulate_one_vm_event() to simply return, doing nothing else. Then,
the guest resumes execution at the same place (since RIP has not been
modified), then hvm_do_resume() gets called again, except this time this
code:

 481     if ( !handle_hvm_io_completion(v) )
 482         return;

ends up trying to MMIO emulate the current instruction by calling
handle_mmio().

I've had hvm_emulate_one_vm_event() return what hvm_emulate_one()
returns, and loop while ( rc == RETRY ) in hvm_do_resume() - this is
functionally equivalent to what I've been doing before by wrapping the
hvm_emulate_one_vm_event() code in a do {} while ( rc == RETRY ); loop
as previously discussed. However, as stated before, at a much later
point I still get rare BSODs, or the guest becomes unresponsive. Digging
a bit more, I've found that what seems to be happening is that emulating
some instructions returns EXCEPTION at some point (for instructions
using the new cmpxchg handler, this can happen if paging_gva_to_gfn()
returns INVALID_GFN and neither PFEC_page_paged nor PFEC_page_shared are
set). Since this causes hvm_emulate_one_vm_event() to
hvm_inject_event(), I'm not sure that it does not mess with the
interrupt part of hvm_do_resume(). This will need more testing to figure
out exactly what's going wrong.


Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel