[Xen-devel] Issue policing writes from Xen to PV domain memory

["Aravindh Puthiyaparambil (aravindp)" <aravindp@xxxxxxxxx>]

This is an issue I am running in to as part of my PV mem_access work. 
http://lists.xenproject.org/archives/html/xen-devel/2014-04/msg03802.html

The code in question is part of this patch:
http://lists.xenproject.org/archives/html/xen-devel/2014-04/msg03805.html

I am running in to an issue when trying to police writes from Xen to the PV 
domain's memory like the the runstate_guest area. If I run the xen-access test 
program to get write events for a PV domain, it causes the host to crash 
immediately after enabling mem_access with the following trace:

(XEN) Assertion 'wqv->esp == 0' failed at wait.c:133
(XEN) ----[ Xen-4.5-unstable  x86_64  debug=y  Tainted:    C ]----
(XEN) CPU:    1
(XEN) RIP:    e008:[<ffff82d08012ffa6>] prepare_to_wait+0x61/0x243
(XEN) RFLAGS: 0000000000010286   CONTEXT: hypervisor
(XEN) rax: 0000000000000100   rbx: 0000000000000100   rcx: ffff82d0802f99cb
(XEN) rdx: ffff82d0802eff00   rsi: 0000000000000000   rdi: ffff83003cc7a000
(XEN) rbp: ffff83003ffe74c8   rsp: ffff83003ffe7498   r8:  000000000001ffff
(XEN) r9:  ffff83003fec0000   r10: 0000000000000030   r11: 0000000000000010
(XEN) r12: ffff83003cc7cb40   r13: ffff830032f44920   r14: ffff83003ffe7f18
(XEN) r15: ffff83003cc7a000   cr0: 000000008005003b   cr4: 00000000000426b0
(XEN) cr3: 0000000012fed000   cr2: ffff88000fc0bd80
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff83003ffe7498:
(XEN)    ffff83003ffe74e4 ffff830032f448e0 ffff830032f44920 ffff830032f45000
(XEN)    ffff83003ffe77a8 0000000000000006 ffff83003ffe7508 ffff82d0801f0ff1
(XEN)    000001003ffe7528 fffffff080243060 00000000000134cb ffff830032f45000
(XEN)    ffff830012ed69e0 ffff830032f45000 ffff83003ffe7528 ffff82d0801f5e78
(XEN)    0000000000000000 ffff83003cc7a000 ffff83003ffe7758 ffff82d08021ae90
(XEN)    ffff830000000005 00000000000333d5 000000000000000b 0000000000000058
(XEN)    ffff82d080319f78 00000000000003f0 0000000000000000 0000000000000880
(XEN)    383283003ffe75d8 0000000000000108 353483003ffe75e8 ffff82d080319f68
(XEN)    000ffff88000fc0b 0000000000000200 00000000000134cb ffff88000fc0bd80
(XEN)    0000000000000201 ffff82d000000000 00000000000134cb 00000000000134cb
(XEN)    623783003ffe7710 ffff82d080313330 ffff83003ffe76f8 ffff82d08012fb27
(XEN)    0030830000000000 ffff82d080363231 ffff830012ed69e0 0000000000000005
(XEN)    0000000000000400 ffff82d0802f99cc 0000000000000001 ffff82d0802f9d7f
(XEN)    ffff83003ffe7770 ffff82d08027358e ffff83003ffe7758 ffff82d0802f9980
(XEN)    0000000000000000 ffff82d0802f9d7f 383583003ffe77a0 ffff82d080333966
(XEN)    ffff83003ffe7788 ffff82d08012fb27 0000000500000000 ffff82d080273590
(XEN)    ffff83003ffe76d8 ffff82d08012f02a 0000000000000400 ffff82d0802f99c8
(XEN)    0000000000000002 ffff82d0802f9d7f ffff83003ffe7800 ffff82d080271e6c
(XEN)    ffff83003ffe77e8 ffff88000fc0bd80 00000000333bd067 00000000333b9067
(XEN)    0000000039f8b067 80100000134cb067 00000000000148fd 00000000000333bd
(XEN) Xen call trace:
(XEN)    [<ffff82d08012ffa6>] prepare_to_wait+0x61/0x243
(XEN)    [<ffff82d0801f0ff1>] __mem_event_claim_slot+0x56/0xb2
(XEN)    [<ffff82d0801f5e78>] mem_access_send_req+0x2e/0x5a
(XEN)    [<ffff82d08021ae90>] sh_page_fault__guest_4+0x1e50/0x2690
(XEN)    [<ffff82d08018dce1>] do_page_fault+0x386/0x532
(XEN)    [<ffff82d08022bc3d>] handle_exception_saved+0x2e/0x6c
(XEN)    [<ffff82d08018defa>] __copy_to_user_ll+0x2a/0x37
(XEN)    [<ffff82d08015fb0f>] update_runstate_area+0xfd/0x106
(XEN)    [<ffff82d08015fb29>] _update_runstate_area+0x11/0x39
(XEN)    [<ffff82d08015fc03>] context_switch+0xb2/0xf03
(XEN)    [<ffff82d080125c23>] schedule+0x5c8/0x5d7
(XEN)    [<ffff82d080128491>] wait+0x9/0xb
(XEN)    [<ffff82d0801f1007>] __mem_event_claim_slot+0x6c/0xb2
(XEN)    [<ffff82d0801f5e78>] mem_access_send_req+0x2e/0x5a
(XEN)    [<ffff82d08021ae90>] sh_page_fault__guest_4+0x1e50/0x2690
(XEN)    [<ffff82d08018dce1>] do_page_fault+0x386/0x532
(XEN)    [<ffff82d08022bc3d>] handle_exception_saved+0x2e/0x6c
(XEN)    [<ffff82d08018defa>] __copy_to_user_ll+0x2a/0x37
(XEN)    [<ffff82d08015fb0f>] update_runstate_area+0xfd/0x106
(XEN)    [<ffff82d08015fb29>] _update_runstate_area+0x11/0x39
(XEN)    [<ffff82d080160a41>] context_switch+0xef0/0xf03
(XEN)    [<ffff82d080125c23>] schedule+0x5c8/0x5d7
(XEN)    [<ffff82d080128a49>] __do_softirq+0x81/0x8c
(XEN)    [<ffff82d080128aa2>] do_softirq+0x13/0x15
(XEN)    [<ffff82d08015cf45>] idle_loop+0x66/0x76
(XEN) 

On adding some debugging, I discovered that it happens after mem_access is 
enabled but xen-access has not started handling events. After comparing the 
stack trace and gla in question There are multiple write faults to the 
runstate_guest(v), each causing an event to be sent to xen-access. Since the 
listener is not handling events yet, the fault continues to occur. I am not 
sure why the listener does not get a chance to run.  I also do not follow is 
that why there are multiple faults as the vcpu should have been paused after 
the first event was sent to xen-access and only be resumed after violation has 
been resolved and when it calls xc_access_resume(), which ends up unpausing the 
vcpu. Or is this occurring because runstate_guest(v).p is being accessed from 
Xen? 

My "fix" for this is to not police writes if they are originating from the Xen 
kernel. I should also mention that while debugging before the hacks went in, if 
I added too much debug to Xen, it would slow things down allowing xen-access to 
start and things would proceed smoothly from there on. In those cases I would 
see multiple consecutive events for runstate_guest area but since they get 
resolved "quick" enough, the crash does not occur.

Please advise as to how I should proceed in solving this issue.

Thanks,
Aravindh


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel