[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] [PATCH RFC] x86/emulate: implement hvmemul_cmpxchg() with an actual CMPXCHG
>>> On 09.04.17 at 00:15, <rcojocaru@xxxxxxxxxxxxxxx> wrote: > On 04/03/2017 09:36 PM, Razvan Cojocaru wrote: >> On 04/03/2017 09:20 PM, Razvan Cojocaru wrote: >>> On 04/01/2017 07:56 PM, Razvan Cojocaru wrote: >>>> On 03/31/2017 06:04 PM, Jan Beulich wrote: >>>>>>>> On 31.03.17 at 17:01, <rcojocaru@xxxxxxxxxxxxxxx> wrote: >>>>>> On 03/31/2017 05:46 PM, Jan Beulich wrote: >>>>>>>>>> On 31.03.17 at 11:56, <rcojocaru@xxxxxxxxxxxxxxx> wrote: >>>>>>>> On 03/31/2017 10:34 AM, Jan Beulich wrote: >>>>>>>>>>>> On 31.03.17 at 08:17, <rcojocaru@xxxxxxxxxxxxxxx> wrote: >>>>>>>>>> On 03/30/2017 06:47 PM, Jan Beulich wrote: >>>>>>>>>>>> Speaking of emulated MMIO, I've got this when the guest was >>>>>>>>>>>> crashing >>>>>>>>>>>> immediately (pre RETRY loop): >>>>>>>>>>>> >>>>>>>>>>>> MMIO emulation failed: d3v8 32bit @ 0008:82679f3c -> f0 0f ba 30 >>>>>>>>>>>> 00 72 >>>>>>>>>>>> 07 8b cb e8 da 4b ff ff 8b 45 >>>>>>>>>>> >>>>>>>>>>> That's a BTR, which we should be emulating fine. More information >>>>>>>>>>> would need to be collected to have a chance to understand what >>>>>>>>>>> might be going one (first of all the virtual and physical memory >>>>>>>>>>> address this was trying to act on). >>>>>>>>>> >>>>>>>>>> Right, the BTR part should be fine, but I think the LOCK part is >>>>>>>>>> what's >>>>>>>>>> causing the issue. I've done a few more test runs to see what return >>>>>>>>>> RETRY (dumping the instruction with an "(r)" prefix to distinguish >>>>>>>>>> from >>>>>>>>>> the UNHANDLEABLE dump), and a couple of instructions return RETRY >>>>>>>>>> (BTR >>>>>>>>>> and XADD, both LOCK-prefixed, which means they now involve CMPXCHG >>>>>>>>>> handler, which presumably now fails - possibly simply because it's >>>>>>>>>> always LOCKed in my patch): >>>>>>>>> >>>>>>>>> Well, all of that looks to be expected behavior. I'm afraid I don't >>>>>>>>> see >>>>>>>>> how this information helps understanding the MMIO emulation failure >>>>>>>>> above. >>>>>>>> >>>>>>>> I've managed to obtain this log of emulation errors: >>>>>>>> https://pastebin.com/Esy1SkHx >>>>>>>> >>>>>>>> The "virtual address" lines that are not followed by any "Mem event" >>>>>>>> line correspond to CMXCHG_FAILED return codes. >>>>>>>> >>>>>>>> The very last line is a MMIO emulation failed. >>>>>>>> >>>>>>>> It's probably important that this happens with the model where >>>>>>>> hvm_emulate_one_vm_event() does _not_ re-try the emulation until it >>>>>>>> succeeds. The other model allows me to go further with the guest, but >>>>>>>> eventually I get timeout-related BSODs or the guest becomes >>>>>>>> unresponsive. >>>>>>> >>>>>>> Interesting. You didn't clarify what the printed "offset" values are, >>>>>>> and it doesn't look like these have any correlation with the underlying >>>>>>> (guest) physical address, which we would also want to see. And then >>>>>>> it strikes me as odd that in these last lines >>>>>>> >>>>>>> (XEN) Mem event (RETRY) emulation failed: d5v8 32bit @ 0008:826bb861 -> >>>>>>> f0 0f > >>>>>> ba 30 00 72 07 8b cb e8 da 4b ff ff 8b 45 >>>>>>> (XEN) virtual address: 0xffd080f0, offset: 4291854576 >>>>>>> (XEN) MMIO emulation failed: d5v8 32bit @ 0008:82655f3c -> f0 0f ba 30 >>>>>>> 00 72 >>>>>> 07 8b cb e8 da 4b ff ff 8b 45 >>>>>>> >>>>>>> the instruction pointers and virtual addresses are different, but the >>>>>>> code bytes are exactly the same. This doesn't seem very likely, so I >>>>>>> wonder whether there's an issue with us wrongly re-using previously >>>>>>> fetched insn bytes. (Of course I'd be happy to be proven wrong with >>>>>>> this guessing, by you checking the involved binary/ies.) >>>>>> >>>>>> Offset is the actual value of the "offset" parameter of >>>>>> hvmemul_cmpxchg(). >>>>> >>>>> That's not very useful then, as for flat segments "offset" == >>>>> "virtual address" (i.e. you merely re-print in decimal what you've >>>>> already printed in hex). >>>> >>>> The attached patch (a combination of your patch and mine) produces the >>>> following output when booting a Windows 7 32-bit guest with monitoring: >>>> https://pastebin.com/ayiFmj1N >>>> >>>> The failed MMIO emulation is caused by a mapping failure due to the >>>> "!nestedhvm_vcpu_in_guestmode(curr) && hvm_mmio_internal(gpa)" condition >>>> being true in hvmemul_vaddr_to_mfn(). I've ripped that off from >>>> __hvm_copy() but it looks like that might not be the right way to use it. >>> >>> Sorry to reply to this email instead of your original reply but I've >>> "left it" in my computer at work. Here's the last part of the log, with >>> the VCPU number logged for the GFN as well: >>> >>> (XEN) [8] gfn: 0x2781 >>> (XEN) [8] virtual address: 0x827810a8, rc: 0 >>> (XEN) [8] gfn: 0x2781 >>> (XEN) [8] virtual address: 0x827810a8, rc: 0 >>> (XEN) [8] gfn: 0x2781 >>> (XEN) [8] virtual address: 0x827810cc, rc: 0 >>> (XEN) [8] gfn: 0x2781 >>> (XEN) [8] virtual address: 0x8278109c, rc: 0 >>> (XEN) [8] gfn: 0x2781 >>> (XEN) [8] virtual address: 0x827810d0, rc: 0 >>> (XEN) [11] gfn: 0x2781 >>> (XEN) [8] gfn: 0x2781 >>> (XEN) [11] virtual address: 0x8278109c, rc: 0 >>> (XEN) [8] virtual address: 0x8278109c, rc: 4 >>> (XEN) Dump follows for VCPU 8 >>> (XEN) Mem event (RETRY) emulation failed: d3v8 32bit @ 0008:826b5c7c -> >>> f0 0f c1 08 85 c9 74 1f f6 c1 02 75 1a 41 8d 41 >>> (XEN) [11] gfn: 0x2781 >>> (XEN) [11] virtual address: 0x827810a8, rc: 0 >>> (XEN) [11] gfn: 0x2781 >>> (XEN) [11] virtual address: 0x827810a8, rc: 0 >>> (XEN) [8] gfn: 0xfed00 >>> (XEN) !page >>> (XEN) hvmemul_vaddr_to_mfn() fail >>> (XEN) [8] virtual address: 0xffd080f0, rc: 1 >>> (XEN) MMIO emulation failed: d3v8 32bit @ 0008:8264ff3c -> f0 0f c1 08 >>> 85 c9 74 1f f6 c1 02 75 1a 41 8d 41 >>> >>> The code does look the same for the last two failures on VCPU 8, for >>> clearly different GFNs and virtual addresses. The first time it hits a >>> protected page, we try to emulate the instruction and it fails with >>> X86EMUL_CMPXCHG_FAILED (rc: 4). Then it somehow pops up again and this >>> time it's MMIO-emulated, and that fails as well (with UNHANDLEABLE, >>> since we can't seem to be able to map the memory). >> >> Another log, making sure that the MMIO emulation fail really happens on >> the same processor as the preceding (identical bytes) fail: >> >> (XEN) [8] gfn: 0x276f >> (XEN) [11] virtual address: 0x8276f09c, rc: 0 >> (XEN) [8] virtual address: 0x8276f09c, rc: 4 >> (XEN) Dump follows for VCPU 8 >> (XEN) Mem event (RETRY) emulation failed: d3v8 32bit @ 0008:826a3861 -> >> f0 0f ba 30 00 72 07 8b cb e8 da 4b ff ff 8b 45 >> (XEN) [11] gfn: 0x276f >> (XEN) [11] virtual address: 0x8276f0d0, rc: 0 >> (XEN) [11] gfn: 0x276f >> (XEN) [11] virtual address: 0x8276f09c, rc: 0 >> (XEN) [11] gfn: 0x276f >> (XEN) [11] virtual address: 0x8276f0cc, rc: 0 >> (XEN) [11] gfn: 0x276f >> (XEN) [11] virtual address: 0x8276f09c, rc: 0 >> (XEN) [8] gfn: 0xfed00 >> (XEN) !page >> (XEN) hvmemul_vaddr_to_mfn() fail >> (XEN) [8] virtual address: 0xffd080f0, rc: 1 >> (XEN) Dump follows for VCPU 8 >> (XEN) MMIO emulation failed: d3v8 32bit @ 0008:8263df3c -> f0 0f ba 30 >> 00 72 07 8b cb e8 da 4b ff ff 8b 45 >> (XEN) [11] gfn: 0x276f >> (XEN) [11] virtual address: 0x8276f0d0, rc: 0 > > As you've suggested, this does indeed seem to happen because of the > current model of emulating because of vm_event replies: an instruction > needs to be emulated in hvm_do_resume(), but inside > hvm_emulate_one_vm_event(), hvm_emulate_one() returns RETRY, presumably > because of the new cmpxchg handler. This causes > hvm_emulate_one_vm_event() to simply return, doing nothing else. Then, > the guest resumes execution at the same place (since RIP has not been > modified), then hvm_do_resume() gets called again, except this time this > code: > > 481 if ( !handle_hvm_io_completion(v) ) > 482 return; > > ends up trying to MMIO emulate the current instruction by calling > handle_mmio(). > > I've had hvm_emulate_one_vm_event() return what hvm_emulate_one() > returns, and loop while ( rc == RETRY ) in hvm_do_resume() - this is > functionally equivalent to what I've been doing before by wrapping the > hvm_emulate_one_vm_event() code in a do {} while ( rc == RETRY ); loop > as previously discussed. However, as stated before, at a much later > point I still get rare BSODs, or the guest becomes unresponsive. Digging > a bit more, I've found that what seems to be happening is that emulating > some instructions returns EXCEPTION at some point (for instructions > using the new cmpxchg handler, this can happen if paging_gva_to_gfn() > returns INVALID_GFN and neither PFEC_page_paged nor PFEC_page_shared are > set). Apparently in line with your later reply - if you got INVALID_GFN from paging_gva_to_gfn(), the earlier read of the same memory should have failed already (unless something plays with the P2M behind your back). Jan > Since this causes hvm_emulate_one_vm_event() to > hvm_inject_event(), I'm not sure that it does not mess with the > interrupt part of hvm_do_resume(). This will need more testing to figure > out exactly what's going wrong. > > > Thanks, > Razvan _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx https://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |