Re: [PATCH V3 (resend) 01/19] x86: Create per-domain mapping of guest_root_pt

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Fri, Jun 14, 2024 at 08:23:30AM +0200, Jan Beulich wrote:
> On 13.06.2024 18:31, Elias El Yandouzi wrote:
> > On 16/05/2024 08:17, Jan Beulich wrote:
> >> On 15.05.2024 20:25, Elias El Yandouzi wrote:
> >>> However, I noticed quite a weird bug while doing some testing. I may
> >>> need your expertise to find the root cause.
> >>
> >> Looks like you've overflowed the dom0 kernel stack, most likely because
> >> of recurring nested exceptions.
> >>
> >>> In the case where I have more vCPUs than pCPUs (and let's consider we
> >>> have one pCPU for two vCPUs), I noticed that I would always get a page
> >>> fault in dom0 kernel (5.10.0-13-amd64) at the exact same location. I did
> >>> a bit of investigation but I couldn't come to a clear conclusion.
> >>> Looking at the stack trace [1], I have the feeling the crash occurs in a
> >>> loop or a recursive call.
> >>>
> >>> I tried to identify where the crash occurred using addr2line:
> >>>
> >>>   > addr2line -e vmlinux-5.10.0-29-amd64 0xffffffff810218a0
> >>> debian/build/build_amd64_none_amd64/arch/x86/xen/mmu_pv.c:880
> >>>
> >>> It turns out to point on the closing bracket of the function
> >>> xen_mm_unpin_all()[2].
> >>>
> >>> I thought the crash could happen while returning from the function in
> >>> the assembly epilogue but the output of objdump doesn't even show the
> >>> address.
> >>>
> >>> The only theory I could think of was that because we only have one pCPU,
> >>> we may never execute one of the two vCPUs, and never setup the mapping
> >>> to the guest_root_pt in write_ptbase(), hence the page fault. This is
> >>> just a random theory, I couldn't find any hint suggesting it would be
> >>> the case though. Any idea how I could debug this?
> >>
> >> I guess you want to instrument Xen enough to catch the top level fault (or
> >> the 2nd from top, depending on where the nesting actually starts) to see
> >> why that happens. Quite likely some guest mapping isn't set up properly.
> >>
> > 
> > Julien helped me with this one and I believe we have identified the 
> > problem.
> > 
> > As you've suggested, I wrote the mapping of the guest root PT in our 
> > per-domain section, root_pt_l1tab, within write_ptbase() function as 
> > we'd always be in the case v == current plus switch_cr3_cr4() would 
> > always flush local tlb.
> > 
> > However, there exists a path, in toggle_guest_mode(), where we could 
> > call update_cr3()/make_cr3() without calling write_ptbase() and hence 
> > not maintain mappings properly. Instead toggle_guest_mode() has a partly 
> > open-coded version of write_ptbase().
> > 
> > Would you rather like to see the mappings written in make_cr3() or in 
> > toggle_guest_mode() within the pseudo open-coded version of write_ptbase()?
> 
> Likely the latter, but that's hard to tell without seeing the resulting
> code.

There's already a special case for XPTI in toggle_guest_mode() to deal
exactly with that AFAICT.  Maybe it would be better if write_ptbase()
could be made suitable to be used in _toggle_guest_pt() instead of
directly calling write_cr3(), as we could then avoid having to pile
open-coded bodges in toggle_guest_mode() and/or _toggle_guest_pt().

Thanks, Roger.