Xen project Mailing List

Re: [Xen-devel] SR-IOV VF Passthrough Issues

To: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>

From: Kaushik Kumar Ram <kaushik@xxxxxxxx>

Date: Thu, 18 Aug 2011 16:14:53 -0500

Delivery-date: Thu, 18 Aug 2011 14:15:59 -0700

List-id: Xen developer discussion <xen-devel.lists.xensource.com>

On Aug 18, 2011, at 9:42 AM, Konrad Rzeszutek Wilk wrote: > On Thu, Aug 18, 2011 at 04:15:02AM -0500, Kaushik Kumar Ram wrote: >> I am trying to passthrough a SR-IOV VF from an Intel 82599EB 10 GbE NIC to a >> PV Linux guest (v3.0.0-rc7). >> I am running a pvops dom0 Linux v2.6.32.44 (xen/stable-2.6.32.x branch). >> >> I followed the instructions on >> http://wiki.xensource.com/xenwiki/XenPCIpassthrough. But it just did not >> work. I got the following error when I tried loading the ixgbevf driver in >> the guest: >> >> ixgbevf 0000:02:10.0: device not available (can't reserve [mem >> 0x00000000-0x00003fff 64bit]) >> ixgbevf: probe of 0000:02:10.0 failed with error -22 >> >> I am not well versed on how PCI devices are setup. But I added some debug >> statements and figured out >> that the device's memory regions were not initialized. I also found that >> passing the 'pci=resource_alignment=' > > Uh, in the dom0? So the lspci showed that region uninitialized? lspci output in dom0 prior to starting the guest. 02:10.0 is the VF I am trying to passthrough to the guest. #> sudo lspci -vv -xxx -s 02:10.0 -n 02:10.0 0200: 8086:10ed (rev 01) Subsystem: 8086:0003 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Region 0: [virtual] Memory at <unassigned> (64-bit, non-prefetchable) Region 3: [virtual] Memory at <unassigned> (64-bit, non-prefetchable) Capabilities: [70] MSI-X: Enable- Count=3 Masked- Vector table: BAR=3 offset=00000000 PBA: BAR=3 offset=00002000 Capabilities: [a0] Express (v0) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 0 ARICtl: MFVC- ACS-, Function Group: 0 Kernel driver in use: pciback 00: ff ff ff ff 00 00 10 00 01 00 00 02 00 00 00 00 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 03 00 30: 00 00 00 00 70 00 00 00 00 00 00 00 00 00 00 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 11 a0 02 00 03 00 00 00 03 20 00 00 00 00 00 00 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a0: 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> argument to dom0 during boot was causing the memory regions to be released >> prior to starting the >> guests. So I just tried removed this particular argument and it worked! Now >> I am able to successfully >> passthrough the VF to the guest. > >> >> So my question is whether the 'pci=resource_alignment=' argument really >> needed? > > It is only needed if the memory region is not page aligned. I wish you > included > the full output of your guest to get an idea of what the BARs and the E820 > looked > like. lspci output in dom0 after starting the guest. Please note that at this point I have already tried loading the ixgbevf driver in the guest and failed. #> sudo lspci -vv -xxx -s 02:10.0 -n 02:10.0 0200: 8086:10ed (rev 01) Subsystem: 8086:0003 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Region 0: [virtual] Memory at <unassigned> (64-bit, non-prefetchable) Region 3: [virtual] Memory at <unassigned> (64-bit, non-prefetchable) Capabilities: [70] MSI-X: Enable- Count=3 Masked- Vector table: BAR=3 offset=00000000 PBA: BAR=3 offset=00002000 Capabilities: [a0] Express (v0) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 0 ARICtl: MFVC- ACS-, Function Group: 0 Kernel driver in use: pciback 00: ff ff ff ff 00 00 10 00 01 00 00 02 00 00 00 00 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 03 00 30: 00 00 00 00 70 00 00 00 00 00 00 00 00 00 00 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 11 a0 02 00 03 00 00 00 03 20 00 00 00 00 00 00 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a0: 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 lspci output in the guest after starting the guest. #> sudo lspci -vv -xxx -s 02:10.0 -n 02:10.0 0200: 8086:10ed (rev 01) Subsystem: 8086:0003 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- Region 0: Memory at <unassigned> (64-bit, non-prefetchable) [disabled] Region 3: Memory at <unassigned> (64-bit, non-prefetchable) [disabled] Capabilities: [70] MSI-X: Enable- Mask- TabSize=3 Vector table: BAR=3 offset=00000000 PBA: BAR=3 offset=00002000 Capabilities: [a0] Express Endpoint IRQ 0 Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag- Device: Latency L0s <64ns, L1 <1us Device: AtnBtn- AtnInd- PwrInd- Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported- Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- Device: MaxPayload 128 bytes, MaxReadReq 128 bytes Link: Supported Speed unknown, Width x0, ASPM unknown, Port 0 Link: Latency L0s <64ns, L1 <1us Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch- Link: Speed unknown, Width x0 Capabilities: [100] Advanced Error Reporting Capabilities: [150] Unknown (14) 00: 86 80 ed 10 00 00 10 00 01 00 00 02 00 00 00 00 10: 04 00 00 00 00 00 00 00 00 00 00 00 04 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 03 00 30: 00 00 00 00 70 00 00 00 00 00 00 00 00 00 00 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 11 a0 02 00 03 00 00 00 03 20 00 00 00 00 00 00 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a0: 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 How do I get hold the E820? Thanks. --Kaushik _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.