Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

[Bruce Edge <bruce.edge@xxxxxxxxx>]

On Wed, Oct 13, 2010 at 3:00 PM, Bruce Edge <bruce.edge@xxxxxxxxx> wrote:
> On Wed, Oct 13, 2010 at 2:46 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@xxxxxxxxxx> wrote:
>> On Wed, Oct 13, 2010 at 02:36:47PM -0700, Bruce Edge wrote:
>>> On Mon, Oct 11, 2010 at 2:46 PM, Konrad Rzeszutek Wilk
>>> <konrad.wilk@xxxxxxxxxx> wrote:
>>> > On Mon, Oct 11, 2010 at 02:12:22PM -0700, Bruce Edge wrote:
>>> >> On Fri, Oct 1, 2010 at 2:11 PM, Konrad Rzeszutek Wilk
>>> >> <konrad.wilk@xxxxxxxxxx> wrote:
>>> >> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
>>> >> >> One of our developers who is working on a tachyon driver is
>>> >> >> complaining that the pvops domU kernel is not working for these MSI
>>> >> >> interrupts.
>>> >> >> This is using the current head of xen/2.6.32.x on both a single
>>> >> >> Nahelam 920 and a dual E5540. This behavior is consistent with Xen
>>> >> >> 4.0.1, 4.0.2.rc1-pre and 4.1.
>>> >> >
>>> >> >
>>> >> > I just checked on my SuperMicro X8DTN, this combination
>>> >> >  - For Dom0, git commit fe999249 (2.6.32.18)
>>> >> >  - For DomU, devel/xen-pcifront-0.6 or devel/xen-pcifront-0.7
>>> >> >  - For Hypervisor I used cs 21976, but found that the latest (22155) 
>>> >> > works too
>>> >> >
>>> >> > with which where I passed in PCI devices with legacy IRQ, MSI, and 
>>> >> > MSI-X. I tried
>>> >> > a combination of doing this with IOMMU (VT-d) and without - both cases 
>>> >> > these devices:
>>> >> >
>>> >> > 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB 
>>> >> > UHCI Controller #1 (rev 02)
>>> >> > 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB 
>>> >> > UHCI Controller #2 (rev 02)
>>> >> > 00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB 
>>> >> > UHCI Controller #3 (rev 02)
>>> >> > 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 
>>> >> > EHCI Controller #1 (rev 02)
>>> >> > 03:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit 
>>> >> > Ethernet Controller (Copper) (rev 06)
>>> >> > 0a:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network 
>>> >> > Connection (rev 02)
>>> >> >
>>> >> > worked just fine (either defining pci=["..."] or just using 
>>> >> > pci-attach).
>>> >> >
>>> >> > But if I use the latest xen/next or xen/stable-2.6.32.x it does not 
>>> >> > look
>>> >> > that happy :-(
>>> >> >
>>> >> >
>>> >>
>>> >> Konrad,
>>> >> To try eliminate the remaining differences here, could you post your
>>> >> dom0/domU config files?
>>> >
>>> > Sure. See attached
>>>
>>> Konrad,
>>> That made a big difference. Looks much better now. It's been kicked
>>> over to several developers who have each got our tachyon driver
>>> working a little bit better.
>>
>> Hmm, I didn't really try to do anything fancy with the configs. Any
>> inklings of what config option might have caused all this headache?
>
> I had pruned out a lot of stuff I didn't need from the kernel. I may
> have been a bit overzealous. Also, we were using the same kernel for
> dom0 and domU. It may be that having all the dom0 baggage in the domU
> has some unexpected consequences. Very vague I know, sorry. If I have
> time I'll try narrow down the diffs that break it.
>
>>
>>>
>>> Now the sticking point is an apparent limitation on the amount of
>>> memory one can request using pci_map_single. It appears that we can
>>> only ask for 256K or less.  We need a 2MB DMA buffer.
>>
>> You can't use SG buffers? And chain them together and provide them
>> to the device? A lot of other drivers do this ...
>
> Perhaps. I don't know. I'm not the driver author. I think that it
> worked using 1 giant buffer before so there was no need to use SG
> buffers.

Got some more info. It's a hardware requirement for the tachyon chip.
It uses a 2 MB block for the FC SEST index table. It's one place where
we can't use SG.
It really not a matter of laziness.

-Bruce

>
>>
>>> Is there some alternate mechanism for getting a larger physically
>>> contiguous buffer under pvops?
>>
>> Why the contiguous requirement?
>
> It may not be a  hard requirement and was just a matter of
> convenience. I'll kick that over the the tachyon guys and see what
> they say. I think it's a non-trivial exercise to switch over. All the
> internal buffer mgmt would have to change. If that's the only option,
> well, that's it. ...is it?
>
> Thanks again for your help. You've been instrumental it getting this up.
>
> -Bruce
>
>>>
>>> Thanks
>>>
>>> -Bruce
>>>
>>> >> I'd like to build the same kernels to get an apples->apples comparison.
>>> >>
>>> >> Also, could you include your grub info and domU cfg file?
>>> >>
>>> >> These may eliminate some of the remaining diffs in the configs and
>>> >> show why your's works while mine does not.
>>> >>
>>> >> Thanks
>>> >>
>>> >> -Bruce
>>> >
>>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel