[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] VT-d async invalidation for Device-TLB.
> From: Zhang, Yang wrote on June 16, 2015 11:07 AM >> Jan Beulich wrote on 2015-06-15: > >>>> On 13.06.15 at 16:44, <quan.xu@xxxxxxxxx> wrote: > >>> On 12.06.15 at 14:47, <JBeulich@xxxxxxxx> wrote: > >>>>>> On 12.06.15 at 04:40, <quan.xu@xxxxxxxxx> wrote: > >>>>>>>>> On 10.06.15 at 16:05, <JBeulich@xxxxxxxx> wrote: > >>>>>>>> On 03.06.15 at 09:49, <quan.xu@xxxxxxxxx> wrote: > >>>>>> For Context Invalidation and IOTLB invalidation without > >>>>>> Device-TLB invalidation, Invalidation Queue flushes synchronous > >>>>>> invalidation as before(This is a tradeoff and the cost of > >>>>>> interrupt is > >>> overhead). > >>>>> > >>>>> DMAR_OPERATION_TIMEOUT being 1s, are you saying that you're not > >>>>> intending to replace the current spinning for the non-ATS case? > >>>> > >>>> Yes, we are not intending to replace the current spinning for the > >>>> non-ATS case. > >>> > >>> I'm not really happy about that. > >> > >> Jan, could you share more about what you expect? We can enable it > >> step by step. > >> Do you want to replace the current spinning for all of queued invalidation? > > > > Yes. > > > >>>>> Considering that expiring these loops results in panic()s, I would > >>>>> expect these to become asynchronous _and_ contained to the > >>>>> affected VM alongside the ATS induced changed behavior. You > >>>>> talking of overhead - can you quantify that? > >>>>> > >>>> > >>>> I tested it by a Myri-10G Dual-Protocol NIC, which is an ATS device. > >>>> for an invalidation: > >>>> By sync way, it takes about 1.4 ms. > >>>> By async way, it takes about 4.3 ms. > > It is a typo for the time: should be 1.4 us and 4.3 us. > yes, it is 1.4 us and 4.3 us. Thanks. I also have tested iotlb sync and async invalidation. For sync way: it takes about 3.2 us. For async way: it takes about 5.3 us. > >>> > >>> What's the theory on why this is? After all, it shouldn't matter how > >>> the completion of the invalidation gets signaled. > >>> > >> > >> Let me introduce more about how I get these test data. > >> > >> For sync way, I get the time by NOW() macro, when I update Queue Tail > >> Register (qinval_update_qtail()). > >> Then get time by NOW() macro in current spinning when it has wrote > >> the value in the Status Data field to the address specified in the Status > Address. > >> The difference of these 2 value is the time of an sync invalidation. > >> > >> > >> For async way, as the previous email, I have introduced the IF bit of > >> Invalidation Wait Descriptor. > >> (IF: Indicate the invalidation wait descriptor completion by > >> generating an invalidation completion event per the programing of the > >> Invalidation Completion Event Registers.) I have implemented an > >> interrupt for invalidation completion event. > >> Also I get the time by NOW() macro, when I update Queue Tail Register > >> (by qinval_update_qtail()). > >> Then get time by NOW() macro in invalidation completion event handler. > >> The difference of these 2 value is the time of an async invalidation. > > > > Okay, thanks for the explanation. As this includes the time it takes > > to deliver and (partly) handle the interrupt, the difference is of > > course within what one would expect (and also of what would seem > acceptable). > > The time doesn't include the cost of handling of interrupt. We just record it > at > the entry of interrupt handler. So the cost should bigger than 4.3 us if > taking the > handing cost into consideration. And the costs will much bigger if there are > more > pass-through VMs runs. We can start from ATS case firstly. And apply it to > non-ATS case later if the async approach doesn't hurt the performance. > Agreed. Now we can implement source code for ATS case first, then we can apply it to no-ATS case later. > > > > Jan > > > Best regards, > Yang Quan _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |