Re: [Xen-devel] Create a iSCSI DomU with disks in another DomU running on the same Dom0

[Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>]

On Sat, Jan 12, 2013 at 01:11:32PM +0100, Roger Pau Monné wrote:
> On 11/01/13 22:09, Konrad Rzeszutek Wilk wrote:
> > On Fri, Jan 11, 2013 at 08:29:12PM +0100, Roger Pau Monné wrote:
> >> On 11/01/13 19:51, Konrad Rzeszutek Wilk wrote:
> >>> On Fri, Jan 11, 2013 at 04:57:52PM +0100, Roger Pau Monné wrote:
> >>>> Hello Konrad,
> >>>>
> >>>> I've found the problem, blkback is adding granted pages to the bio that 
> >>>> is then passed to the underlying block device. When using a iscsi 
> >>>> target running on another DomU in the same h/w this bios end up in 
> >>>> netback, and then when performing the gnttab copy operation, it 
> >>>> complains because the passed mfn belongs to a different domain.
> >>>
> >>> OK, so my original theory was sound. The m2p override "sticks".
> >>>>
> >>>> I've checked this by applying the appended patch to blkback, which 
> >>>> allocates a buffer to pass to the bio instead of using the granted 
> >>>> page. Of course this should not applied, since it implies additional 
> >>>> memcpys.
> >>>>
> >>>> I think the right way to solve this would be to change netback to 
> >>>> use gnttab_map and memcpy instead of gnttab_copy, but I guess this 
> >>>> will imply a performance degradation (haven't benchmarked it, but I 
> >>>> assume gnttab_copy is used in netback because it is faster than 
> >>>> gnttab_map + memcpy + gnttab_unmap).
> >>>
> >>> Or blkback is altered to use grant_copy.
> >>
> >> This would not work with the persistent-grants extension, and also when
> >> scaling to a large number of guests will probably have a degraded
> >> performance due to the grant table lock (compared to using persistent
> >> grants).
> >>
> >>> Or perhaps m2p_override
> >>> can do multiple PAGE_FOREIGN? (So if it detects a collision it will
> >>> do something smart.. like allocate a new page or update the 
> >>> kmap_op with extra information).
> >>
> >> What we could do is add extra information to m2p_override, containing
> >> the grant_ref_t and domid, so when a FOREIGN_FRAME is detected in
> >> grant_copy (or netback) the grant_ref_t and domid of the passed mfn is
> >> used instead of the mfn (provided that grant_copy can perform a copy
> >> between two grant references of different domains).
> 
> Since the issue I'm having is not common I'm not sure if this solution
> is worth it, it will imply storing a pointer to a struct in the page
> private data, that stores the mfn, grant reference and domid (now we are
> only storing the mfn in the page private data).
> 
> >>>
> >>>
> >>> And yes, grant_map in netback is much much slower that grant_copy
> >>> (I tested 2.6.32 vs 3.7 using a Xen 4.1.3 with the grant_copy fixes
> >>> that Jan came up with).
> >>
> >> Yes, I see there's no way we are going to use grant_map instead of
> >> grant_copy. I guess this will no longer be true once netback/front
> >> starts using the persistent grants extension.
> > 
> > Hm? Annie posted patches for the persistent grants on netback/netfront
> > and they did not show much improvement (as you are already doing 
> > grant_copy).
> > 
> > Or are you saying change netback to use grant_map and utilize the
> > skb->deconstructor to keep track of it? And then do persistent grant
> > extensions on it?
> 
> I'm not really familiar with the net code, but I've had a quick look at
> the grant_copy operation in Xen and it is indeed using the grant lock to
> protect some parts of the code.

Sure, but at the cost of doing memory copy. And if the guests are on 
seperate sockets there are no cache benefits.

> 
> Using persistent grants should provide better performance in the long
> run because once the grant is mapped we don't have to issue any more
> grant operations, thus avoiding grant lock contention (maybe I'm missing
> something here). skb deconstructor should probably be used in netfront,

They did not improve it. As a matter of fact they made it worst.
(So this is taking the idea that Andrew had that he shared with Oliver and you
about doing persistent grants, and using the same type of hypercalls and
copy - but do it in the network subsystem).

> to return the persistently mapped grant to the list of free grants, but
> I'm not sure if we will need to use it in netback.

Why not? We don't want to blow any the cache data if we can do it. The
problem is how to deal with the RX path from the NICs. Each NICs on the
RX path does something like this:

        1). process its descriptors
        2). unmap the page
        3). allocate a new set of skbs (and pages), update the descriptors
            with the new bus address
        4). pass off the unmapped pages (with skbs) to the network stack.
        5). forget about the skbs.

There is no persistency - so the NIC ends up getting a "fresh" set of pages
all the time that perculate up to netback.

The TX (so netback -> NIC) could be solved by having a pool of pages/skb's
that are mapped and are owned by netback (via the skb->dcst) so that they
are retained.
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel