Re: [Xen-devel] Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))

["Jan Beulich" <JBeulich@xxxxxxxx>]

>>> On 11.12.13 at 22:30, Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx> wrote:
> On Wed, Dec 11, 2013 at 09:15:17PM +0000, Gordan Bobic wrote:
>> On 12/11/2013 06:32 PM, Konrad Rzeszutek Wilk wrote:
>> >Interestingly enough I just hit this with my brand-new Haswell CPU and
>> >new motherboard when passing in a capture card. It shows:
>> >
>> >     +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree 
>> > Corporation Bt878 Video 
> Capture
>> >            |                               |            +-08.1  Brooktree 
> Corporation Bt878 Audio Capture
>> >            |                               |            +-09.0  Brooktree 
> Corporation Bt878 Video Capture
>> >            |                               |            +-09.1  Brooktree 
> Corporation Bt878 Audio Capture
>> >            |                               |            +-0a.0  Brooktree 
> Corporation Bt878 Video Capture
>> >            |                               |            +-0a.1  Brooktree 
> Corporation Bt878 Audio Capture
>> >            |                               |            +-0b.0  Brooktree 
> Corporation Bt878 Video Capture
>> >            |                               |            \-0b.1  Brooktree 
> Corporation Bt878 Audio Capture
>> >            |                               \-03.0  Texas Instruments 
> TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
>> >
>> >And Xen says:
>> >(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
>> >(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
>> >(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] 
>> >fault 
> addr 36aa3000, iommu reg = ffff82c3ffd53000
>> >(XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
>> >(XEN) print_vtd_entries: iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 36aa3
>> >(XEN)     root_entry = ffff83083d47e000
>> >(XEN)     root_entry[8] = 72569a001
>> >(XEN)     context = ffff83072569a000
>> >(XEN)     context[0] = 0_0
>> >(XEN)     ctxt_entry[0] not present
>> >(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
>> >(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
>> >(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] 
>> >fault 
> addr 36aa3000, iommu reg = ffff82c3ffd53000
>> >
>> >
>> >Oddly enough it was working fine in a box with an AMD IOMMU. But
>> >to be fair - that machine was running with Xen 4.1.
>> >
>> >The hack I developed: 
> http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html 
>> >ends up with this:
>> >
>> >(XEN) alloc_pdev: unknown type: 0000:08:00.0
>> >(XEN) [VT-D]iommu.c:1484: d0:unknown(0): 0000:08:00.0
>> >(XEN) [VT-D]iommu.c:1888: d0: context mapping failed
>> >
>> >(FYI, this Xen 4.3.1)
>> >
>> >Let me retry on the AMD box with the same version of Xen.
>> 
>> I may be wrong, but this doesn't look like the same problem (phantom
>> PCI device on the bus). Or am I missing something?
> 
> It is. A phantom device as well.

Nothing in what you posted confirms this (because it doesn't
show what the upstream bridge(s) is/are).

>> As far as I can tell, the original problem was arising on cards that
>> are PCIe, but based on a PCIX chipset, i.e. with a PCIe-PCIX bridge.
>> Xen wasn't the only thing affected in my case - bare metal Linux
>> kernel was also having problems with intel-iommu=1 in the kernel
>> boot parameters. If might be worth trying that with your card to see
>> what happens. If bare metal Linux with intel-iommu=1 works for your
>> card, it's probably not the same problem (of course it could be
>> similar/related).
> 
> That is a similar problem here. Except that I have a PCI devices and
> it goes over an PCIe bridge (I think).

As said above - to see that's the case would require to see more
lspci output than what you provided.

>> Out of interest, I noticed recently there is a xen parameter
>> "pci-phantom", but I haven't been able to find documentation for it.
>> Can you point me in the right direction? Does it, perchance, allow
>> specifying the PCI slot ID of a phantom device so that IOMMU doesn't
>> freak out when a seemingly non-existant device starts trying to do
>> DMA?
> 
> I forgot about it!
> 
> t 4e3c592c93d7dbe02ca36878457515d30fe931d2
> Author: Jan Beulich <jbeulich@xxxxxxxx>
> Date:   Mon Jan 7 12:58:09 2013 +0100
> 
>     IOMMU: add option to specify devices behaving like ones using phantom 
> functions

Note the term "functions" here: This is about a feature of the PCI
spec that some devices apparently use without declaring they do.
This has nothing to do with entire devices being invisible, in order
for there to be phantom functions there always has to be an
ordinary device at function 0.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel