RE: [Xen-devel] MSI and VT-d interrupt remapping

["Jiang, Yunhong" <yunhong.jiang@xxxxxxxxx>]

xen-devel-bounces@xxxxxxxxxxxxxxxxxxx <> wrote:
> [Keir Fraser]
>> On 6/3/08 21:07, "Espen Skoglund"
> <espen.skoglund@xxxxxxxxxxxxx> wrote:
>>> If an MSI capable device was able to make use of the above feature,
>>> the device could be set up to generate different interrupts
>>> depending on where the incoming interrupt was to be handled.  For
>>> example, incoming data for a particular guest could trigger an
>>> interrupt on the processor where that guest is running.  Obviously,
>>> a dom0-like backend driver would not be involved in the actual
>>> event delivery in these situations.  The event would be delivered
>>> directly to the frontend. 
>>> 
>>> The necessary changes would enable a device driver for an MSI
capable
>>> device to allocate a range of pirqs and bind these to different
>>> frontends.
> 
>> The only tricky bit here is deciding what the interface should be to
>> the hypervisor to specify these allocation constraints.
> 
>> Another thought though: there's no good reqson for Xen to scatter
>> its irq-vector allocations across the vector space. That's a
>> holdover from classic-Pentium-era systems, which could lose
>> interrupts if too many got 'queued up' at any single priority
>> level. So we could actually allocate our vectors contiguously,
>> making it much more likely that you could successfully allocate a
>> contiguous range even without remapping.
> 
>> However, I guess you want to be able to specify different target
>> APICs for different vectors too, so again it comes back to: what
>> should the guest interface to irq-remapping hardware be?
> 
> Right.  The reason for bringing up this suggestion now rather than
> later is because MSI support has not yet found its way into mainline.
> Whoever decides on the interface used for registering MSI and MSI-X
> interrupts might want to take multi-message MSIs into account as well.

Espen, thanks for your comments. I remember Linux has not such support,
so Linux driver will not benifit from such implementation. After all,
driver need provide ISR for the interrupts. Of course, we need this
feature if any OS has support. I didn't support this because it may
require changes to various common components and need more discussion,
while Linux has no support to it. (also I rushed to 3.2 cut-off at that
time :$).

> 
> I do not think explicitly specifying destination APIC upon allocation
> is the best idea.  Setting the affinity upon binding the interrupt
> like it's done today seems like a better approach.  This leaves us
> with dealing with the vectors.

But what should happen when the vcpu is migrated to another physical
cpu? I'm not sure the cost to program the interrupt remapping table,
otherwise, that is a good choice to achieveh the affinity.

As for vector assignment, I agree simpler method is to change the vector
assignment in xen as Keir suggestion. Also I suspect we may need support
per-CPU vector later, if there are so many vector requested.

> 
> My initial thought was to make use of the new msix_entries[] field in
> the xen_pci_op structure.  This field is already used as an in/out
> parameter for allocating MSI-X interrupts.  The pciback_enable_msi()
> function can then attempt to allocate multiple interrupts instead of a
> single one, and return the allocated vectors.
> 
> The current MSI patchset also lacks a set_affinity() function for
> changing the APIC destination similar to what is done for, e.g.,
> IOAPICs.  Also similar to IOAPICs, the MSI support should have
> something like the io_apic_write_remap_rte() for rewriting the
> interrupt remapping table when enabled.

For the set_affinity(),  what do you mean of changing the APIC
destination? Currently if set guest's pirq's affinity, it will only
impact event channel. The physical one will only be called once, when
the pirq is bound.

As for rewriting interrupt remapping table like
io_apic_write_remap_rte(), I think it will be added later also.

I'm also a bit confused for your statement in previous mail "The
necessary changes would enable a device driver for an MSI capable device
to allocate a range of pirqs and bind these to different frontends.".
What do you mean of different frontends?

Really thanks for your suggestion.

> 
> A special case must exist when setting the interrupt affinity for
> multiple-message enabled MSI devices.  There probably should exist
> some magic in the set_affinity() function for handling this properly.
> That is, setting affinity for the whole group of MSI interrupts does
> not make all that much sense.  It makes more sense when one can set
> the per-interrupt affinity through the interrupt remapping table.
> 
> It should be evident by now that my suggestions for deciding upon an
> interface and implementation of it is rather fluffy; borderlining
> non-existent.  This is partly because I'm talking about not yet
> existing MSI support, but mainly because I'm still new to Xen
> internals.  Nonetheless, I believe it would be good for people working
> on MSI support to take multi-message and interrupt remapping into
account
> as well. 
> 
>       eSk
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel