Re: [Xen-ia64-devel] Re: PMT table for XEN/IA64 (was: RE: Transparentparavirtualization vs. xen paravirtualization)

[Matt Chapman <matthewc@xxxxxxxxxxxxxxx>]

I'm still not clear about the details.  Could you outline the changes
that you want to make to Xen/ia64?

Would DomU have a PMT?  Surely DomU should not know about real machine
addresses, that should be hidden behind the grant table interface.
Otherwise migration, save/restore, etc. are difficult (as they have
found on x86).

Do you know how x86 shadow_translate mode works?  Perhaps we should
use that as an example.

Matt


On Mon, Oct 31, 2005 at 05:11:09PM +0800, Tian, Kevin wrote:
> Matt Chapman wrote:
> > 1. Packet arrives in a Dom0 SKB.  Of course the buffer needs
> >    to be page sized/aligned (this is true on x86 too).
> > 2. netback steals the buffer
> > 3. netback donates it to DomU *without freeing it*
> > 4. DomU receives the frame and passes it up its network stack
> > 5. DomU gives away other frame(s) to restore balance
> > 6. Dom0 eventually receives extra frames via its balloon driver
> > 
> > 5 and 6 can be done lazily in batches.  Alternatively, 4 and 5
> > could be a single "flip" operation.
> 
> The solution will work with some tweaks.  But is there any obvious benefit 
> than PMT approach used on x86? (If yes, you should suggest to xen-devel;-) 
> Usually we want a different approach for either "can't do on this 
> architecture" or "far better performance than existing one". Or else why we 
> derail from Xen design for extra maintainance effort.  This extra effort has 
> causing us 2+ weeks to get VBD up to support DomU for the last 2 upstream 
> merges
> 
> > 
> > I think this is not significantly different from x86.
> > 
> > I'm not saying this is necessarily better than a PMT solution,
> > but I want to discuss the differences and trade-offs.  By PMT
> > I assume you mean to make Dom0 not 1:1 mapped, and then give
> > it access to the translation table?  Can you describe how the
> > above works differently with a PMT?
> 
> 
> Simply saying the work flow, PMT approach is similar with backend/frontend 
> needed to touch PMT table for ownership change. However do you evaluate how 
> many tricky changes required to support Domain0 with gpn=mfn upon existing 
> code? For example,
>       - Backend drivers are not bound to dom0, which can also be used by domU 
> as driver domain. At that time, 1:1 mapping has no sense there.  There are 
> some talks on DomU servers as driver IO already.
>       - You need ensure all available pages granted to dom0. That means you 
> need change current dom0 allocation code. 
>       - You need to change current vnif code with - unknown - #ifdefs and 
> workarounds, since you implement a new behavior on top of different approach.
>       - ... (maintenance!)
> 
> So if you implement a VM from scratch, then definitely your approach is 
> worthy of trying since no limitation there. However since we work on XEN, we 
> should take advantage of current Xen design as possible, right? ;-)
> 
> > 
> > One disadvantage I see of having Dom0 not 1:1 is that superpages
> > are more difficult, we can't just use the guest's superpages.
> 
> 
> Superpages are optimization option, and we still need to support incontiguous 
> pages as a basic requirement. You can still add option to allocate contiguous 
> pages for guest even with PMT table, since para-virtualization is cooperative.
> 
> > 
> > Also, are there paravirtualisation changes needed to support a
> > PMT?  I'm concerned about not making the paravirtualisation
> > changes too complex (I think x86 Xen changes the OS too much).
> > Also, it should be possible to load Xen frontend drivers into
> > unmodified OSs (on VT).
> 
> 
> We need balance between new designs and maintainance effort. Currently 
> Xiaofeng Lin from Intel is working on para-drivers for unmodified domain, and 
> both VBD & VNIF are working for x86 VT domains already and are reviewing by 
> Cambridge. This work is based on PMT table.
> 
> Kevin
> > 
> > On Mon, Oct 31, 2005 at 01:28:43PM +0800, Tian, Kevin wrote:
> >> Hi, Matt,
> >> 
> >>    The point here is how to check donated frame done and where "free"
> >> actually happens in domU. Currently Linux network driver utilizes
> >> zero-copy to pass received packet up without any copy. In this case,
> >> the receive pages are allocated from skbuff, which however is freed
> >> by upper layer instead of vnif driver itself. To let dom0 know when
> >> the donated page is done, you may either:     
> >>    - Copy content from donated page to local skbuff page, and then
> >> notify dom0 immediately at the cost of performance 
> >>    - Modify upper layer code to register "free" hook which notify dom0
> >> if done at the cost of more modification to common code and bias
> >> from x86.  
> >> 
> >>    Definitely there're other possibilities to make it "working" by
> >> this approach and even more alternatives. However the point we
> >> really want to emphasize here is that we can move towards x86
> >> solution by adding PMT, with best performance and less maintenance
> >> effort. That can actually minimize our future re-base effort when
> >> para-drivers keep going. ;-)     
> >> 
> >> Thanks,
> >> Kevin
> >> 
> >>> -----Original Message-----
> >>> From: Matt Chapman [mailto:matthewc@xxxxxxxxxxxxxxx]
> >>> Sent: 2005å10æ31æ 13:09
> >>> To: Tian, Kevin
> >>> Cc: Dong, Eddie; xen-ia64-devel@xxxxxxxxxxxxxxxxxxx
> >>> Subject: Re: [Xen-ia64-devel] Re: PMT table for XEN/IA64 (was: RE:
> >>> Transparentparavirtualization vs. xen paravirtualization)
> >>> 
> >>> Yes, I think I understand the problem now.
> >>> 
> >>> The way I imagine this could work is that Dom0 would know about all
> >>> of the memory in the machine (i.e. it would be passed the original
> >>> EFI memmap, minus memory used by Xen).
> >>> 
> >>> Then Dom0 would donate memory for other domains (=ballooning).
> >>> Dom0 can donate data frames to DomU in the same way - by granting
> >>> the frame and not freeing it.  When DomU donates a data frame to
> >>> Dom0, Dom0 frees it when it is done, and now the kernel can use it.
> >>> 
> >>> What do you think of this approach?
> >>> 
> >>> Matt
> >>> 
> >>> 
> >>> On Mon, Oct 31, 2005 at 11:09:04AM +0800, Tian, Kevin wrote:
> >>>> Hi, Matt,
> >>>>  It's not related to mapped virtual address, but only for
> >>>> physical/machine pfn. 
> >>> Current vnif backend (on x86) works as:
> >>>> 
> >>>> 1. Allocate a set of physical pfns from kernel
> >>>> 2. chop up the mapping between physical pfn and old machine pfn
> >>>> 3. Transfer ownership of old machine pfn to frontend
> >>>> 4. Allocate new machine pfn and bound to that physical pfn
> >>>> (In this case, there's no ownership return from frontend for
> >>>> performance reason) 
> >>>> 
> >>>>  If without PMT table (Assuming guest==machine for dom0), that
> >>>> means you 
> >>> have to hotplug physical pfns from guest (based on page
> >>> granularity) based on current vnif model. Or maybe you have better
> >>> alternative without PMT, and without big change to existing vnif
> >>> driver simultaneously? 
> >>>> 
> >>>> Thanks,
> >>>> Kevin
> >>>> 
> >>>>> -----Original Message-----
> >>>>> From: xen-ia64-devel-bounces@xxxxxxxxxxxxxxxxxxx
> >>>>> [mailto:xen-ia64-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of
> >>>>> Matt Chapman Sent: 2005å10æ31æ 10:59 To: Dong, Eddie
> >>>>> Cc: xen-ia64-devel@xxxxxxxxxxxxxxxxxxx
> >>>>> Subject: [Xen-ia64-devel] Re: PMT table for XEN/IA64 (was: RE:
> >>>>> Transparentparavirtualization vs. xen paravirtualization)
> >>>>> 
> >>>>> Hi Eddie,
> >>>>> 
> >>>>> The way I did it was to make the address argument to grant
> >>>>> hypercalls in/out; that is, the hypervisor might possibly return
> >>>>> a different address than the one requested, like mmap on UNIX.
> >>>>> 
> >>>>> For DomU, the hypervisor would map the page at the requested
> >>>>> address.  For Dom0, the hypervisor would instead return the
> >>>>> existing address of that page, since Dom0 already has access
> >>>>> to the whole address space.
> >>>>> 
> >>>>> (N.B. I'm referring to physical/machine mappings here; unlike
> >>>>> the x86 implementation where the grant table ops map pages
> >>>>> directly into virtual address space).
> >>>>> 
> >>>>> Matt
> >>>>> 
> >>>>> 
> >>>>> On Fri, Oct 28, 2005 at 10:28:08PM +0800, Dong, Eddie wrote:
> >>>>>>>  Page flipping should work just fine
> >>>>>>> in the current design; Matt had it almost working (out of tree)
> >>>>>>> before he went back to school.
> >>>>>>> 
> >>>>>> Matt:
> >>>>>>        Dan mentioned that you had VNIF work almost done without PMT
> >>>>>> table support for dom0, Can you share the idea with us?
> >>>>>>        Usually VNIF swap page between dom0 and domU so that network
> >>>>>> package copy (between dom0 native driver and domU  frontend
> >>>>>> driver) can be avoided and thus achieve high performance. With
> >>>>>> this swap, we can no longer assume dom0 gpn=mfn. So what did you
> >>>>>>        ever propose to port VNIF without PMT table? Thanks a lot, eddie
> >>>>> 
> >>>>> _______________________________________________
> >>>>> Xen-ia64-devel mailing list
> >>>>> Xen-ia64-devel@xxxxxxxxxxxxxxxxxxx
> >>>>> http://lists.xensource.com/xen-ia64-devel
> > 
> > _______________________________________________
> > Xen-ia64-devel mailing list
> > Xen-ia64-devel@xxxxxxxxxxxxxxxxxxx
> > http://lists.xensource.com/xen-ia64-devel

_______________________________________________
Xen-ia64-devel mailing list
Xen-ia64-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-ia64-devel