Re: [Xen-devel] [PATCH 0 of 3] Avoid sharing vectors within a device when using an AMD IOMMU

[George Dunlap <george.dunlap@xxxxxxxxxx>]

On Tue, 2011-07-26 at 18:01 +0100, Keir Fraser wrote:
> On 26/07/2011 17:33, "George Dunlap" <george.dunlap@xxxxxxxxxxxxx> wrote:
> 
> > The interrupt remapping tables on AMD IOMMUs index by vector only.
> > This means that if two MSIs go through the table that are destined for
> > different cpus, but they share the same vector, they will be
> > redirected to the same place.  (E.g., one interrupt on p5 vector 67,
> > another interrupt on p7 vector 67; both will be redirected to the same
> > place.)
> > 
> > Introducing per-device interrupt mappings reduces the problem, but
> > does not solve it completely if the same device can have multiple IRQs
> > assigned to it, because you can get the same issue -- two different
> > IRQs from the same device can be assigned the same vector on different
> > cpus.  This causes one of the IRQs to activated when either interrupt
> > is triggered, and the other IRQ to never receive any interrupts.
> 
> The patches look fine, but I don't see a reason to add yet more command-line
> options. AMD systems using irq remapping will need to avoid vector sharing;
> other systems do not need to do so. We detect that automatically and DTRT
> and there's no reason for a user to override that. It's just extending the
> set of arcane IOMMU command line settings that noone will understand the
> implications of specifying.

I was mainly thinking of the support issue.  We don't do extensive
testing of AMD passthrough here at Citrix (or we probably would have
tripped over this problem sooner).  If there are subtle, unexpected
side-effects of this patch, being able to tell people, "Specify
iommu=no-perdevice-irq-map" is a lot easier than telling them to install
a custom build with the perdevice stuff off.  (And it's something that
power users can share with other users without needing to be a
developer.)

I'm not sure how common it is to have devices register more than one
interrupt.  If it's just specialist multi-queue cards, then it may be
the case that having per-device intremap tables will suffice in many
cases.  If nearly all devices have multiple interrupts, then you're
right, the option to disable it for AMDs with IOMMUs disabled is pretty
pointless.

 -George


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel