Re: [Xen-devel] [XenSummit 2017] Notes from the 5-level-paging session

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 20/07/17 11:36, Yu Zhang wrote:
> On 7/20/2017 6:42 PM, Andrew Cooper wrote:
> > On 20/07/17 11:10, Yu Zhang wrote:
> > > On 7/17/2017 6:53 PM, Juergen Gross wrote:
> > > > Hey,
> > > >
> > > > I took a few notes at the 5-level-paging session at the summit.
> > > > I hope there isn't any major stuff missing...
> > > >
> > > > Participants (at least naming the active ones): Andrew Cooper,
> > > > Jan Beulich, Yu Zhang and myself (the list is just from my memory).
> > > >
> > > > The following topics have been discussed in the session:
> > > >
> > > >
> > > > 1. Do we need support for 5-level-paging PV guests?
> > > >
> > > > There is no urgent need for 5-level-paging PV guests for the
> > > > following reasons:
> > > >
> > > > - Guests >64TB (which is the upper limit for 4-level-paging Linux)
> > > >    can be PVH or HVM.
> > > >
> > > > - A 5-level-paging host supports up to 4 PB physical memory. A
> > > >    4-level-paging PV-Dom0 can support that theoretically: the M2P map
> > > >    for 4 PB memory needs 8 TB space, which just fits into the 
> > > > hypervisor
> > > >    reserved memory area in the Linux kernel. Any other hypervisor data
> > > >    and/or code can live in the additionally available virtual space of
> > > >    the 5-level-paging mode.
> > > >
> > > > There was agreement we don't need support of 5-level-paging PV guests
> > > > right now. There is a need, however, to support 4-level-paging PV
> > > > guests located anywhere in the 52-bit physical space of a 
> > > > 5-level-paging
> > > > host (right now they would have to be in the bottom 64 TB as the Linux
> > > > kernel is masking away any MFN bit above 64 TB). I will send 
> > > > patches to
> > > > support this.
> > > >
> > > >
> > > > 2. Do we need 5-level-paging shadow mode support?
> > > >
> > > > While strictly required for PV guests only and no 5-level-paging PV
> > > > guests are to be supported, we will need 5-level-paging shadow mode in
> > > > the long run. This is necessary because even for a 4-level-paging PV
> > > > guest (or a 32-bit PV guest) the processor will run in 5-level-paging
> > > > mode on a huge host as switching between the paging modes is rather
> > > > complicated and should be avoided. It is much easier to run shadow
> > > > mode for the whole page table tree instead for two subtrees only.
> > > >
> > > > OTOH the first step when implementing 5-level-paging in the hypervisor
> > > > doesn't require shadow mode to be working, so it can be omitted in the
> > > > beginning.
> > > >
> > > >
> > > > 3. Is it possible to support 5-level-paging in Xen via a specific
> > > >     binary for the first step?
> > > >
> > > > Yu Zhang asked for implementing 5-level-paging via a Kconfig option
> > > > instead of dynamical switching at boot time for the first prototype.
> > > > This request was accepted in order to reduce the complexity of the
> > > > initial patches. Boot time switching should be available for the
> > > > final solution, though.
> > > >
> > > >
> > > > I hope I didn't miss anything.
> > > Thanks a lot for the your help and for the summary, Juergen.
> > > And I really need to say thank you for quite a lot people who joined 
> > > this discussion. It's quite
> > > enlightening. :)
> > >
> > > One thing I can recall is about the wr{fs,gs}base for pv guest. 
> > > IIRC, our agreement is to turn off
> > > the FSGSBASE in cr4 for PV guests and try to emulate the 
> > > rd{fs,gs}base and wr{fs,gs}base in the
> > > #UD handler.
> > >
> > > But please correct me if I misunderstood. :)
> > Yes, that matches my understanding.
> >
> > A second piece of emulation which needs to happen is to modify the 
> > #PF handler to notice if a PV guest takes a fault with %cr2 being 
> > va57 canonical but not va48 canonical.  In this case, we need to 
> > decode the instruction as far as working out the segment of the 
> > memory operand, and inject #GP[0]/#SS[0] as appropriate.
> Thanks, Andrew. So working out the segment is only to decide if #GP or 
> #SS is to be injected, right?
Correct.  Any memory reference with an explicit %ss override, or which 
uses %rsp/%rbp as a base register needs to be #SS.  Everything else is #GP.
> And I'm wondering, even when pv guest and hypervisor are both running 
> in 4 level paging mode,
> it could be possible for a #PF to have a va48 canonical address, but 
> there's no #GP/#SS injected.
> So it is left to the PV guest kernel I guess?
Most pagefaults get handled either by Xen, or by Xen deeming that the 
pagefault was caused by the guest, and passing the pagefault on to the 
guest kernel.  Xen doesn't really care at this point; it is the guest 
kernel's job to work out what the correct next action is.
> And if the answer is yes, in 5 level case, to whom shall we inject the 
> fault? PV guest kernel shall not
> handle this fault, right?
The problem we need to fix is new with Xen running in 5-level.

Previously with Xen running in 4-level, any non-va48 canonical address 
would yield #GP/#SS and Xen would handle these directly (usually, by 
passing them off the to guest kernel like we do with #PF).
When Xen is running in 5-level, a 64bit PV guest still running in 
4-levels is can actually use memory references in the va57 canonical 
range, because the hardware is actually operating in 5 levels.
In the context of a 64bit PV guest, the vast majority of the va57 range 
will be not-present (as Xen is handling the L5 table on behalf of the 
unaware PV guest), while the area Xen resides in will be mapped 
supervisor and take a #PF that way.
If we were to do the simple thing and hand the #PF to the guest kernel, 
that would be architecturally wrong, because the guest kernel is 
expecting to be running in 4 levels.  Therefore, Xen needs to emulate 
the difference by converting the #PF to #GP/#SS for the guest kernel, so 
the behaviour as observed by the guest kernel matches what is expected 
from 4-levels.
I hope this is a clearer way of explaining the problem.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel