|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
On Fri, Dec 13, 2013 at 03:27:25PM +0000, Gordan Bobic wrote:
> On 12/13/2013 02:56 PM, Jan Beulich wrote:
> >>>>On 13.12.13 at 15:43, Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
> >>>>wrote:
> >>-[0000:00]-+-00.0 Intel Corporation 4th Gen Core Processor DRAM Controller
> >> +-01.0-[01]--+-00.0 Intel Corporation 82571EB Gigabit Ethernet
> >> Controller
> >> | \-00.1 Intel Corporation 82571EB Gigabit Ethernet
> >> Controller
> >> +-01.1-[02]----00.0 LSI Logic / Symbios Logic SAS2008
> >> PCI-Express Fusion-MPT SAS-2 [Falcon]
> >> +-02.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core
> >> Processor Integrated Graphics Controller
> >> +-03.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core
> >> Processor HD Audio Controller
> >> +-14.0 Intel Corporation 8 Series/C220 Series Chipset Family
> >> USB xHCI
> >> +-16.0 Intel Corporation 8 Series/C220 Series Chipset Family
> >> MEI Controller #1
> >> +-16.3 Intel Corporation 8 Series/C220 Series Chipset Family
> >> KT Controller
> >> +-19.0 Intel Corporation Ethernet Connection I217-LM
> >> +-1a.0 Intel Corporation 8 Series/C220 Series Chipset Family
> >> USB EHCI #2
> >> +-1b.0 Intel Corporation 8 Series/C220 Series Chipset High
> >> Definition Audio Controller
> >> +-1c.0-[03]----00.0 Intel Corporation 82574L Gigabit Network
> >> Connection
> >> +-1c.1-[04]----00.0 Intel Corporation 82574L Gigabit Network
> >> Connection
> >> +-1c.3-[05]----00.0 Intel Corporation I210 Gigabit Network
> >> Connection
> >> +-1c.4-[06]----00.0 Intel Corporation 82572EI Gigabit Ethernet
> >> Controller (Copper)
> >> +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0 Brooktree
> >> Corporation Bt878 Video Capture
> >> | | +-08.1 Brooktree
> >> Corporation Bt878 Audio Capture
> >> | | +-09.0 Brooktree
> >> Corporation Bt878 Video Capture
> >> | | +-09.1 Brooktree
> >> Corporation Bt878 Audio Capture
> >> | | +-0a.0 Brooktree
> >> Corporation Bt878 Video Capture
> >> | | +-0a.1 Brooktree
> >> Corporation Bt878 Audio Capture
> >> | | +-0b.0 Brooktree
> >> Corporation Bt878 Video Capture
> >> | | \-0b.1 Brooktree
> >> Corporation Bt878 Audio Capture
> >
> >So 00:1c.5 is the PCIe->PCI bridge, 07:00.0 and 08:01.0 are
> >PCI-PCI bridges, the capture devices are ordinary ones. The only
> >possibly unexpected aspect I can see here is that there are two
> >intermediate PCI-PCI bridges, but iirc this should be handled by
> >the VT-d code.
>
> Not sure about multiple intermediate PCI-PCI bridges, but I can
> confirm that multiuple intermediate PCIe bridge setup works fine,
> e.g. Intel X58 -> Nvidia NF200 -> PLX -> GPU (GTX690)
To double-check whether this is a problem with the card itself,
I plucked the card in an AMD box (TA890FXE) and an Intel box (ThinkServer
TS130) and they work great there. It only has one PCIe->PCI bridge
for the PCI cards.
TA890FXE:
00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
(prog-if 01 [Subtractive decode])
Flags: bus master, 66MHz, medium devsel, latency 64
Bus: primary=00, secondary=02, subordinate=02, sec-latency=64
I/O behind bridge: 0000c000-0000dfff
02:05.0 PCI bridge: Hint Corp HB6 Universal PCI-PCI bridge (non-transparent
mode) (rev 11) (prog-if 00 [Normal decode])
Flags: bus master, medium devsel, latency 64
Bus: primary=02, secondary=03, subordinate=03, sec-latency=64
Prefetchable memory behind bridge: fde00000-fdefffff
Capabilities: [80] Power Management version 2
Capabilities: [90] CompactPCI hot-swap <?>
+-14.4-[02-03]--+-05.0-[03]--+-08.0 Brooktree Corporation Bt878 Video
Capture
| | +-08.1 Brooktree Corporation Bt878
Audio Capture
| | +-09.0 Brooktree Corporation Bt878
Video Capture
| | +-09.1 Brooktree Corporation Bt878
Audio Capture
| | +-0a.0 Brooktree Corporation Bt878
Video Capture
| | +-0a.1 Brooktree Corporation Bt878
Audio Capture
| | +-0b.0 Brooktree Corporation Bt878
Video Capture
| | \-0b.1 Brooktree Corporation Bt878
Audio Capture
| \-06.0 NetMos Technology PCI 9835 Multi-I/O
Controller
ThinkServer TS130:
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5) (prog-if 01
[Subtractive decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=02, subordinate=03, sec-latency=32
Prefetchable memory behind bridge: 00000000d0000000-00000000d00fffff
Capabilities: [50] Subsystem: Lenovo Device 1025
00:00.0 Host bridge: Intel Corporation Device 0108 (rev 09)
Subsystem: Lenovo Device 1025
Flags: bus master, fast devsel, latency 0
Capabilities: [e0] Vendor Specific Information: Len=0c <?>
+-1e.0-[02-03]----00.0-[03]--+-08.0 Brooktree Corporation Bt878
Video Capture
| +-08.1 Brooktree Corporation Bt878
Audio Capture
| +-09.0 Brooktree Corporation Bt878
Video Capture
| +-09.1 Brooktree Corporation Bt878
Audio Capture
| +-0a.0 Brooktree Corporation Bt878
Video Capture
| +-0a.1 Brooktree Corporation Bt878
Audio Capture
| +-0b.0 Brooktree Corporation Bt878
Video Capture
| \-0b.1 Brooktree Corporation Bt878
Audio Capture
SuperMicro (two PCI bridgeS):
+-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0 Brooktree
Corporation Bt878 Video Capture
| | +-08.1 Brooktree
Corporation Bt878 Audio Capture
| | +-09.0 Brooktree
Corporation Bt878 Video Capture
| | +-09.1 Brooktree
Corporation Bt878 Audio Capture
| | +-0a.0 Brooktree
Corporation Bt878 Video Capture
| | +-0a.1 Brooktree
Corporation Bt878 Audio Capture
| | +-0b.0 Brooktree
Corporation Bt878 Video Capture
| | \-0b.1 Brooktree
Corporation Bt878 Audio Capture
| \-03.0 Texas Instruments TSB43AB22A
IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI
Express Root Port #6 (rev d4) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=07, subordinate=09, sec-latency=0
Memory behind bridge: f0400000-f05fffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805
Capabilities: [a0] Power Management version 3
Kernel driver in use: pcieport
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor
PCI Express x16 Controller (rev 06) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 0000e000-0000efff
Memory behind bridge: f0d00000-f0dfffff
Capabilities: [88] Subsystem: Intel Corporation Device 2010
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [a0] Express Root Port (Slot+), MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [140] Root Complex Link
Capabilities: [d94] #19
Kernel driver in use: pcieport
07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01) (prog-if 01
[Subtractive decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=07, secondary=08, subordinate=09, sec-latency=32
Memory behind bridge: f0400000-f05fffff
Capabilities: [60] Subsystem: Super Micro Computer Inc Device 0805
Capabilities: [a0] Power Management version 3
Which would look like this:
C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs) on the card
\--------------> IEEE-1394a
I am actually wondering if this 07:00.0 device is the one that
reports itself as 08:00.0 (which I think is what you alluding to Jan)
Gordan (in earlier emails) reported that the VT-d does work for him and
he has much more complex setup than I do on this motherboard:
This is on http://www.newegg.com/Product/Product.aspx?Item=N82E16813188070
\-[0000:00]-+-00.0 Intel Corporation 5520 I/O Hub to ESI Port [8086:3406]
+-07.0-[05-08]----00.0-[06-08]--+-00.0-[08]--+-00.0 NVIDIA
Corporation GF100GL [Quadro 6000] [10de:06d8]
| | \-00.1 NVIDIA
Corporation GF100 High Definition Audio Controller [10de:0be5]
| \-02.0-[07]--+-00.0 NVIDIA
Corporation GF100GL [Quadro 5000] [10de:06d9]
| \-00.1 NVIDIA
Corporation GF100 High Definition Audio Controller [10de:0be5]
00:07.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express
Root Port 7 [8086:340e] (rev 22) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Bus: primary=00, secondary=05, subordinate=08, sec-latency=0
I/O behind bridge: 0000c000-0000dfff
Memory behind bridge: f4000000-fbdfffff
05:00.0 PCI bridge [0604]: NVIDIA Corporation NF200 PCIe 2.0 switch [10de:05b1]
(rev a3) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Bus: primary=05, secondary=06, subordinate=08, sec-latency=0
I/O behind bridge: 0000c000-0000dfff
Memory behind bridge: f4000000-fbdfffff
Prefetchable memory behind bridge: 00000000a8000000-00000000bfffffff
06:00.0 PCI bridge [0604]: NVIDIA Corporation NF200 PCIe 2.0 switch [10de:05b1]
(rev a3) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Bus: primary=06, secondary=08, subordinate=08, sec-latency=0
I/O behind bridge: 0000d000-0000dfff
Memory behind bridge: f8000000-fbdfffff
Prefetchable memory behind bridge: 00000000b4000000-00000000bfffffff
Thought I am having a hard time parsing the 'lspci -vt' I think it is:
X58---> NF200 ---> Quadro 6000 (GPU and audio card)
\---> NF200 ---> Quadro 5000 (GPU and audio card).
Which is similar setup as mine - an intermediate bridge.
So I think the code is OK - this is likely a firmware/motherboard
bug.
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |