[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] VT-d spin loops
On 26/06/14 11:30, Jan Beulich wrote: > All, > > VT-d code currently has a number of cases where completion of certain > operations is being waited for by way of spinning. The various instances > can be identified relatively easily by grep-ing for all uses of > DMAR_OPERATION_TIMEOUT (the majority of instances use that > variable indirectly through IOMMU_WAIT_OP()), allowing for loops of > up to 1 second. While in many of the cases this _may_ be acceptable > (which would need to be proven for each individual case, also taking into > consideration how many of these spinning loops may be executed in a > row with no preemption/scheduling in between), the invalidation case > seems particularly problematic: Using DMAR_OPERATION_TIMEOUT is > a mistake here in the first place, as the timeout here isn't related to > response times by the IOMMU engine. Instead - with ATS in use - the > specification mandates a timeout of 1 _minute_ (with a 50% slack, the > meaning of which none of us [Andrew and Malcolm brought this issue > to my attention in private talks on the hackathon] was able to really > interpret in a sensible way). From a worst case point of view, this 50% slack means 90 seconds total. > > So there are two things that need doing rather sooner than later: > > First and foremost the ATS case needs to be taken into consideration > when doing invalidations. Obviously we can't spin for a minute, so > invalidation absolutely needs to be converted to a non-spinning model. > We realize this isn't going to be trivial, which is why we bring this up > here rather than coming forward with a patch right away. > > Second, looking at Linux (which interestingly enough also spins, and > that even without any timeout) there are flags in the fault status > register that can be used to detect at least some loop abort conditions. > We should definitely make use of anything that can shorten these > spinning loops (as was already done in commit dd6d87a4 ["VT-d: drop > redundant calls to invalidate_sync()"] as a very tiny first step). The > main problem with trying to clone at least some of the functionality > from Linux is that I'm not convinced the replaying they do can > actually do anything good. Plus it's clear that - spinning or not - the > consequences of an invalidation request not completing successfully > need to be taken care of (and it's of no help that in all cases I looked > at so far errors passed up from the leaf functions sooner or later > get dropped on the floor or mis-interpreted). > > And finally, all other spinning instances need to be audited to make > sure they can't add up to multiple-second spins (iirc we can't > tolerate more than about 4s without running into time problems on > certain hardware). > > Jan > With a default watchdog, the maximum possible spin time is 5 seconds. However, given the 1 second time calibration rendezvous, any spin longer than 1 second will hold all other Xen cpus up waiting for the spinning cpu, which is a stall of the entire results in a stall of the entire system until the spinner has exited. ~Andrew _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |