Re: [Xen-devel] Re: Possible regression in "x86-64: reduce range spanned by 1:1 mapping and frame table indexes"

[Simon Horman <horms@xxxxxxxxxxxx>]

On Mon, Dec 07, 2009 at 11:13:34PM +1100, Simon Horman wrote:
> On Mon, Dec 07, 2009 at 11:06:44AM +0000, Jan Beulich wrote:
> > >>> Simon Horman <horms@xxxxxxxxxxxx> 07.12.09 11:48 >>>
> > >On Mon, Dec 07, 2009 at 10:37:43AM +0000, Jan Beulich wrote:
> > >> While I can't determine the exact source location corresponding to the
> > >> crash (without the disassembly of the function), the page table walk
> > >> suggests this is a read from to the M2P table, which imposes a couple
> > >> of questions: How can this be a non-quad-word aligned access? Is the
> > >> access, if it makes sense, guarded by an mfn_valid() check? Is the
> > >> memory address corresponding to the M2P slot (mfn 0x3c20001) in a
> > >> physical memory hole?
> > >
> > >Any tips on how I could investigate those questions?
> > 
> > For the first two, you'd have to connect register/stack values to
> > source variables (by analyzing the disassembly) to understand what
> > access it is that causes the issue, and where the values come from.
> > Or alternatively just add debugging printk()-s to the function in
> > question (but that could be a lot of output depending on how long the
> > guest survives). Or whatever else debugging technique you like...
> 
> Thanks, I was fearing something along those lines.
> 
> > For the third, all it takes is looking up the memory map in the hypervisor
> > (boot) log.
> 
> Ok, thats an easy one :-)
> 
> I'll poke some more tomorrow.

Hi Jan,

I haven't exactly followed your advice above, but I think I have
made a little bit of progress.

* The memory access is to a MMIO region - a control register for the NIC.
  The access is made by the e1000 gPXE driver.

* Interestingly, if there is a write to the page (or perhaps even anywhere
  in the entire 20-page MMIO region?) before a read, the problem doesn't
  occur. So a simple work-around in the gPXE driver is to just write to a
  register before reading any.

* In sh_page_fault() the call to gfn_to_mfn_guest_dbg(d, gfn, &p2mt) sets
  p2mt to p2m_mmio_direct, which seems to be correct. I'm a bit
  stuck in working out what goes wrong from there.

For reference, a panic with some more debug info is below.

106697 sh: sh_page_fault__guest_2(): fast path mmio 0x000000000b8f9a
106698 sh: sh_page_fault__guest_2(): d:v=1:0 va=0xb8f9c err=15, rip=7a39
106699 sh: sh_page_fault__guest_2(): 2954: A
106700 sh: sh_page_fault__guest_2(): 2998: B
106701 sh: sh_page_fault__guest_2(): fast path mmio 0x000000000b8f9c
106702 sh: sh_page_fault__guest_2(): d:v=1:0 va=0xb8f9e err=15, rip=7a39
106703 sh: sh_page_fault__guest_2(): 2954: A
106704 sh: sh_page_fault__guest_2(): 2998: B
106705 sh: sh_page_fault__guest_2(): fast path mmio 0x000000000b8f9e
106706 sh: sh_page_fault__guest_2(): d:v=1:0 va=0xf30000d8 err=2, rip=6f6
106707 sh: sh_page_fault__guest_2(): 2954: A
106708 sh: sh_page_fault__guest_2(): 2998: B
106709 sh: sh_page_fault__guest_2(): 3077: page_fault_slow_path:
106710 sh: sh_page_fault__guest_2(): 3081: C
106711 sh: sh_page_fault__guest_2(): 3095: rewalk:
106712 sh: sh_page_fault__guest_2(): 3097: D
106713 sh: sh_page_fault__guest_2(): 3106: E
106714 sh: sh_page_fault__guest_2(): 3122: F
(XEN) _gfn_to_mfn_type_dbg: current
106715 sh: sh_page_fault__guest_2(): 3141: gfn_to_mfn_guest_dbg: p2mt=5
106716 sh: sh_page_fault__guest_2(): 3143: G
106717 sh: sh_page_fault__guest_2(): 3181: H
106718 sh: sh_page_fault__guest_2(): 3193: I
106719 sh: sh_page_fault__guest_2(): 3204: J
106720 sh: sh_page_fault__guest_2(): 3216: K
106721 shdebug: make_fl1_shadow(): (f3000)=>118644
106722 sh: set_fl1_shadow_status(): gfn=f3000, type=00000002, smfn=118644
106723 shdebug: _sh_propagate(): demand write level 2 guest f30000e7 shadow 
0000000118644067
106724 sh: sh_page_fault__guest_2(): 3241: L
106725 shdebug: _sh_propagate(): demand write level 1 guest f3000067 shadow 
00000000f0500037
106726 sh: sh_page_fault__guest_2(): 3263: M
106727 shdebug: _sh_propagate(): prefetch level 1 guest f3001067 shadow 
00000000f0501037
106728 shdebug: _sh_propagate(): prefetch level 1 guest f3002067 shadow 
00000000f0502037
106729 shdebug: _sh_propagate(): prefetch level 1 guest f3003067 shadow 
00000000f0503037
106730 shdebug: _sh_propagate(): prefetch level 1 guest f3004067 shadow 
00000000f0504037
106731 shdebug: _sh_propagate(): prefetch level 1 guest f3005067 shadow 
00000000f0505037
106732 shdebug: _sh_propagate(): prefetch level 1 guest f3006067 shadow 
00000000f0506037
106733 shdebug: _sh_propagate(): prefetch level 1 guest f3007067 shadow 
00000000f0507037
106734 shdebug: _sh_propagate(): prefetch level 1 guest f3008067 shadow 
00000000f0508037
106735 shdebug: _sh_propagate(): prefetch level 1 guest f3009067 shadow 
00000000f0509037
106736 shdebug: _sh_propagate(): prefetch level 1 guest f300a067 shadow 
00000000f050a037
106737 shdebug: _sh_propagate(): prefetch level 1 guest f300b067 shadow 
00000000f050b037
106738 shdebug: _sh_propagate(): prefetch level 1 guest f300c067 shadow 
00000000f050c037
106739 shdebug: _sh_propagate(): prefetch level 1 guest f300d067 shadow 
00000000f050d037
106740 shdebug: _sh_propagate(): prefetch level 1 guest f300e067 shadow 
00000000f050e037
106741 shdebug: _sh_propagate(): prefetch level 1 guest f300f067 shadow 
00000000f050f037
106742 shdebug: _sh_propagate(): prefetch level 1 guest f3010067 shadow 
00000000f0510037
106743 shdebug: _sh_propagate(): prefetch level 1 guest f3011067 shadow 
00000000f0511037
106744 shdebug: _sh_propagate(): prefetch level 1 guest f3012067 shadow 
00000000f0512037
106745 shdebug: _sh_propagate(): prefetch level 1 guest f3013067 shadow 
00000000f0513037
106746 shdebug: _sh_propagate(): prefetch level 1 guest f3014067 shadow 
00000000f0514037
106747 shdebug: _sh_propagate(): prefetch level 1 guest f3015067 shadow 
00000000f0515037
106748 shdebug: _sh_propagate(): prefetch level 1 guest f3016067 shadow 
00000000f0516037
106749 shdebug: _sh_propagate(): prefetch level 1 guest f3017067 shadow 
00000000f0517037
106750 shdebug: _sh_propagate(): prefetch level 1 guest f3018067 shadow 
00000000f0518037
106751 shdebug: _sh_propagate(): prefetch level 1 guest f3019067 shadow 
00000000f0519037
106752 shdebug: _sh_propagate(): prefetch level 1 guest f301a067 shadow 
00000000f051a037
106753 shdebug: _sh_propagate(): prefetch level 1 guest f301b067 shadow 
00000000f051b037
106754 shdebug: _sh_propagate(): prefetch level 1 guest f301c067 shadow 
00000000f051c037
106755 shdebug: _sh_propagate(): prefetch level 1 guest f301d067 shadow 
00000000f051d037
106756 shdebug: _sh_propagate(): prefetch level 1 guest f301e067 shadow 
00000000f051e037
106757 shdebug: _sh_propagate(): prefetch level 1 guest f301f067 shadow 
00000000f051f037
106758 sh: sh_page_fault__guest_2(): 3285: N
106759 sh: sh_page_fault__guest_2(): 3310: O
106760 sh: sh_page_fault__guest_2(): 3319: P
106761 sh: sh_page_fault__guest_2(): 3332: Q
106762 sh: sh_page_fault__guest_2(): 3343: goto emulate;
106763 sh: sh_page_fault__guest_2(): 3361: emulate:
106764 sh: sh_page_fault__guest_2(): 3367: R
106765 sh: sh_page_fault__guest_2(): 3390: emulate_readonly:
106766 sh: sh_page_fault__guest_2(): 3403: early_emulation:
106767 sh: sh_page_fault__guest_2(): 3405: S
106768 sh: sh_page_fault__guest_2(): emulate: eip=0x6f6 esp=0x3d264
106769 sh: sh_page_fault__guest_2(): 3446: T
106770 sh: sh_page_fault__guest_2(): emulator failure, unshadowing mfn 0xf0500
106771 sh: sh_remove_shadows(): d=1, v=0, gmfn=f0500
(XEN) ----[ Xen-3.5-unstable  x86_64  debug=y  Tainted:    C ]----
(XEN) CPU:    1
(XEN) RIP:    e008:[<ffff82c4801c6cfa>] sh_remove_shadows+0x169/0x922
(XEN) RFLAGS: 0000000000010282   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: ffff8300ded42000   rcx: 000000000000000a
(XEN) rdx: 00000000000003f8   rsi: 0000000000000282   rdi: ffff82c480235c64
(XEN) rbp: ffff830118fe7b58   rsp: ffff830118fe7b28   r8:  0000000000000000
(XEN) r9:  ffff82c480201820   r10: 00000000ffffffff   r11: 0000000000000005
(XEN) r12: ffff82f601e0a000   r13: 00000000000f0500   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 00000000000026f0
(XEN) cr3: 00000001164d3000   cr2: ffff82f601e0a00f
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff830118fe7b28:
(XEN)    0000000100000001 0000000000000001 ffff8301164e0000 ffff8300ded42000
(XEN)    0000000000000000 ffff830118fe7f28 ffff830118fe7e18 ffff82c4801d3497
(XEN)    00000000000006f6 0000000000002c98 ffff830118fe7f28 ffff830118fe7f28
(XEN)    ffff82c480265b80 ffff830118fe7f28 ffff830118fe7f28 ffff830118fe7f28
(XEN)    ffff830118fe7f28 ffff830118fe7dc8 ffff830118fe7cc8 ffff830118fe7f28
(XEN)    ffff830118fe7ce0 ffff830118fe7f28 00000000f30000d8 ffff8301164e0eb0
(XEN)    ffff8301164e0e18 ffff8301164e0210 ffff818000798000 0000004300000000
(XEN)    0000000000000000 0000000000000f30 00000000000f3000 00000000000f0500
(XEN)    00000000000000d8 ffff8300ded43a38 ffff8140c0003cc0 0000000000000030
(XEN)    ffffffffffffffff 00000020f81dcee8 ffff8180007980f8 000000000000001f
(XEN)    000000000000c39e ffffffffffffffff 00000000000f3000 0000000000118644
(XEN)    ffff818000798100 00000000000f3000 00000000000f3000 00000000000f3000
(XEN)    0000000000000000 0000002018fe7de8 00000000000f0500 0000000000000000
(XEN)    00000000f051f037 ffff8300ded42000 00000000001164cf 0000000500000005
(XEN)    ffff830118fe7f28 0000002000000020 ffff830118fe0000 ffff000000d880c7
(XEN)    e405f608408bffff b3ff1275010003c2 1f9335680000011c 00000000000006f6
(XEN)    0c9f000800000007 07ec2ce0ffffffff 0c93001000000000 07ec2ce0ffffffff
(XEN)    0c93001000000000 07ec2ce0ffffffff 0000c09300000000 0000000000000005
(XEN)    ffff830118fe7de8 ffff830118fe7f28 000000000000000a ffff82c48023d5c0
(XEN)    ffff830118fe7dc8 fffffffffffffffe ffff8300ded42000 ffff830118fe7de8
(XEN) Xen call trace:
(XEN)    [<ffff82c4801c6cfa>] sh_remove_shadows+0x169/0x922
(XEN)    [<ffff82c4801d3497>] sh_page_fault__guest_2+0x1f2d/0x23fb
(XEN)    [<ffff82c4801b83cd>] vmx_vmexit_handler+0x716/0x19b4
(XEN)    
(XEN) Pagetable walk from ffff82f601e0a00f:
(XEN)  L4[0x105] = 00000000decfa027 5555555555555555
(XEN)  L3[0x1d8] = 000000011bffb063 5555555555555555
(XEN)  L2[0x00f] = 0000000000000000 ffffffffffffffff 
(XEN) debugtrace_dump() starting
(XEN) debugtrace_dump() finished
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 1:
(XEN) FATAL PAGE FAULT
(XEN) [error_code=0000]
(XEN) Faulting linear address: ffff82f601e0a00f
(XEN) ****************************************
(XEN) 
(XEN) Manual reset required ('noreboot' specified)


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel