Re: [Xen-devel] Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))

[Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>]

On Tue, Jan 07, 2014 at 10:44:29AM +0000, Gordan Bobic wrote:
> On 2014-01-07 10:38, Andrew Cooper wrote:
> >On 07/01/14 10:35, Gordan Bobic wrote:
> >>On 2014-01-07 03:17, Zhang, Yang Z wrote:
> >>>Konrad Rzeszutek Wilk wrote on 2014-01-07:
> >>>>>Which would look like this:
> >>>>>
> >>>>>C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs)
> >>>>>on the card
> >>>>>          \--------------> IEEE-1394a
> >>>>>
> >>>>>I am actually wondering if this 07:00.0 device is the one that
> >>>>>reports itself as 08:00.0 (which I think is what you alluding to
> >>>>>Jan)
> >>>>>
> >>>>
> >>>>And to double check that theory I decided to pass in the IEEE-1394a
> >>>>to a guest:
> >>>>
> >>>>           +-1c.5-[07-08]----00.0-[08]----03.0  Texas Instruments
> >>>>TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
> >>>>
> >>>>
> >>>>(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow (XEN)
> >>>>[VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN)
> >>>>[VT-D]iommu.c:865: DMAR:[DMA Read] Request device
> >>>>[0000:08:00.0] fault
> >>>>addr 370f1000, iommu reg = ffff82c3ffd53000 (XEN) DMAR:[fault reason
> >>>>02h] Present bit in context entry is clear (XEN) print_vtd_entries:
> >>>>iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 370f1 (XEN)
> >>>>root_entry
> >>>>= ffff83083d47f000 (XEN)     root_entry[8] = 72569b001 (XEN)
> >>>>context
> >>>>= ffff83072569b000 (XEN)     context[0] = 0_0 (XEN)
> >>>>ctxt_entry[0]
> >>>>not present
> >>>>
> >>>>So, capture card OK - Likely the Tundra bridge has an issue:
> >>>>
> >>>>07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
> >>>>(prog-if 01 [Subtractive decode])
> >>>>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> >>>>        ParErr- Stepping- SERR- FastB2B- DisINTx- Status:
> >>>>Cap+ 66MHz-
> >>>>        UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+
> >>>>        >SERR- <PERR- INTx- Latency: 0 Bus: primary=07,
> >>>>secondary=08,
> >>>>        subordinate=08, sec-latency=32 Memory behind bridge:
> >>>>        f0600000-f06fffff Secondary status: 66MHz+ FastB2B+ ParErr-
> >>>>        DEVSEL=medium TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> >>>>BridgeCtl:
> >>>>        Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
> >>>>                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> >>>>        Capabilities: [60] Subsystem: Super Micro Computer Inc
> >>>>Device 0805
> >>>>        Capabilities: [a0] Power Management version 3
> >>>>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> >>>>                PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0
> >>>>NoSoftRst+
> >>>>                PME-Enable- DSel=0 DScale=0 PME-
> >>>>
> >>>>or there is some unknown bridge in the motherboard.
> >>>
> >>>According your description above, the upstream Linux should also have
> >>>the same problem. Did you see it with upstream Linux?
> >>
> >>The problem I was seeing with LSI cards (phantom device doing DMA)
> >>does, indeed, also occur in upstream Linux. If I enable intel-iommu on
> >>bare metal Linux, the same problem occurs as with Xen.
> >>
> >>>There may be some buggy device that generate DMA request with
> >>>internal
> >>>BDF but it didn't expose it(not like Phantom device). For those
> >>>devices, I think we need to setup the VT-d page table manually.
> >>
> >>I think what is needed is a pci-phantom style override that tells the
> >>hypervisor to tell the IOMMU to allow DMA traffic from a specific
> >>invisible device ID.
> >>
> >>Gordan
> >
> >There is.  See "pci-phantom" in
> >http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html
> 
> I thought this was only applicable to phantom _functions_ (number
> after the
> dot) rather than whole phantom _devices_. Is that not the case?

My new Supermicro X10SAE has this issue as well and I wrote a patch to 
"fix" this. Just to be clear - this is an issue with the motherboard
(or rather the PCI-to-PCI bridge) is that it violates VT-d chapter
3.9.2:
"For devices behind conventional PCI bridges, the source-id in the DMA
requests is the requestor-id of the bridge device.".

In my case the bridge is is:

06:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)

But all the requests from the devices behind the bridge show up with
request-id +1 (aka 07:00.0):


(XEN) [2014-02-05 22:23:23] [VT-D]iommu.c:880: iommu_fau3] [VT-D]iommu.c:882: 
iommu_fault_status: Primary Pending Fault
(XEN) [2014-02-05 22:23:23] [VT-D]iommu.c:860: DMAR:[DMA Read] Request device 
[0000:07:00.0] fault addr a57fc000, iommu reg = ffff82c000203000
(XEN) [2014-02-05 22:23:23] DMAR:[fault reason 02h] Present bit in context 
entry is clear
(XEN) [2014-02-05 22:23:23] print_vtd_entries: iommu ffff83023948b9c0 dev 
0000:07:00.0 gmfn a57fc
(XEN) [2014-02-05 22:23:23]     root_entry = ffff82004000d000
(XEN) [2014-02-05 22:23:23]     root_entry[7] = 1b8eba001
(XEN) [2014-02-05 22:23:23]     context = ffff82004000e000
(XEN) [2014-02-05 22:23:23]     context[0] = 0_0
(XEN) [2014-02-05 22:23:23]     ctxt_entry[0] not present
[   21.305708] firewire_ohci 0000:07:03.0: added OHCI v1.10 device as card 0, 4 
IR + 8 IT contexts, quirks 0x2
(XEN) [2014-02-05 22:23:24] [VT-D]iommu.c:880: iommu_fault_status: Fault 
Overflow
(XEN) [2014-02-05 22:23:24] [VT-D]iommu.c:882: iommu_fault_status: Primary 
Pending Fault
(XEN) [2014-02-05 22:23:24] [VT-D]iommu.c:860: DMAR:[DMA Read] Request device 
[0000:07:00.0] fault addr b76cd000, iommu reg = ffff82c000203000
(XEN) [2014-02-05 22:23:24] DMAR:[fault reason 02h] Present bit in context 
entry is clear
(XEN) [2014-02-05 22:23:24] print_vtd_entries: iommu ffff83023948b9c0 dev 
0000:07:00.0 gmfn b76cd
(XEN) [2014-02-05 22:23:24]     root_entry = ffff82004000d000
(XEN) [2014-02-05 22:23:24]     root_entry[7] = 1b8eba001
(XEN) [2014-02-05 22:23:24]     context = ffff82004000e000
(XEN) [2014-02-05 22:23:24]     context[0] = 0_0
(XEN) [2014-02-05 22:23:24]     ctxt_entry[0] not present
[   21.386993] firewire_ohci 0000:07:03.0: bad self ID 0/1 (00000000 != 
~00000000)
[   21.402892] initcall fw_ohci_init+0x0/0x1000 [firewire_ohci] returned 0 
after 254567 usecs


Anyhow, the "solution" was to provide a "link" to the device
being passed in and create a fake device in Xen so that when
I passthrough my fireware: 07:03.0 it will also setup
a context for 07:00.0 device (which does not exist at all).

Attached is the Linux kernel module I use to let the hypervisor
know about the new device (this could have been written as
an user application too).

And the patch for the hypervisor, along with some extra debug
patches so when doing 'xl debug-keys Q' you can get a better
sense of what is what.

The 0004-xen-pci-Introduce-a-way-to-deal-with-buggy-hardware-.patch
is what is interesting.

If there is an interest in upstream this I can take a look -
but I will need guidance from Jan how he would like to do it.

Attachment: hack.c
Description: Text document

Attachment: 0004-xen-pci-Introduce-a-way-to-deal-with-buggy-hardware-.patch
Description: Text document

Attachment: 0001-pci-On-PCI-dump-keyhandler-include-bus2bridge-inform.patch
Description: Text document

Attachment: 0002-pci-On-PCI-dump-device-keyhandler-include-Device-and.patch
Description: Text document

Attachment: 0003-DEBUG-Include-upstream-bridge-information.patch
Description: Text document

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel