Re: [Xen-users] Problem with PCI Pass-through address space collision

[Gordan Bobic <gordan@xxxxxxxxxx>]

I'm pretty sure I seem to recall that PCI passthrough will not work 
without VT-d, but by all means, feel free to try.
Even if you did have working VT-d, though, you have to detach the device 
from dom0 before you can add it to domU, using something like:
virsh nodedev-detach pci_0000_06_01_0

Given the EL6 CRC Xen packages you are using, they use pciback built as 
a module, so kernel boot parameters won't help. What you need to do is 
add this to /etc/modprobe.d/:
# cat xen-pciback.conf
options xen-pciback permissive=1 hide=(06:01.0)

Run depmod -a once you have done that.

Then:

# modprobe xen-pciback
virsh nodedev-detach pci_0000_06_01_0

Also add the driver for the card to /etc/modprobe.d/blacklist.conf.

After that you should be able to boot the domU with the device.

You may also want to upgrade to the latest testing packages (4.2.2-5) 
since they include a PCI passthrough fix from a couple of days ago, 
although it doesn't look like you are falling foul of it.
Also, how much RAM are you passing to domU? Try giving it <= 2GB. There 
is a PCI memory map bug that can cause a nasty memory stomp that kept me 
chasing my tail for days. For most people it manifests at > 4GB, but on 
my system it manifested at > 2GB.
HTH.

Gordan

On 05/20/2013 06:04 PM, Jon Skilling wrote:
> Hi,
>
> I’ve been trying to configure Xeon on my HP ML350 G4 server for the past
> two weeks and despite reading just about every word of the Xen wiki and
> numerous other posts and mails, I can’t find a solution to my problem.
> Any help on this would be much appreciated!
>
> Setup:
>
> HP ML350 G4, Dual xeon, 6Gb Ram, 6 disk scsi raid array,  Digium TDM410P
> analogue PBX card on PCI.  Hardware virtualization (Vt-d) is not an
> option with this machine.
>
> I followed these instructions (more or less) to set up Dom0 and DomU:
>
> http://www.howtoforge.com/virtualization-with-xen-on-centos-6.3-x86_64-paravirtualization-and-hardware-virtualization
>
> with the following changes:
>
> Host Dom0 (Centos 6.4):
>
> xen-4.2.2-4.el6.x86_64
>
> kernel-xen-3.9.2-1.el6xen.x86_64
>
> libvirt 1.0.3-1 (python-virtinstall causes libvirt to be upgraded to
> 1.0.3. From checking the source, the Xen patch appears to be there
> already, so no recompile needed – the Xen patch doesn’t work with this
> source anyway.
>
> XEND has been disabled from boot up because it causes problems with XL
> tools although the same address space collision occurs if I use the XM
> tool set.
>
> I have tried xen-pciback.hide(06:01.0)  on the kernel module definitions
> in boot.conf but this doesn’t seem to do anything. Adding records to
> modprobe.conf and rc.local work better.
>
> The device I’m trying to passthrough is defined:
>
> 06:01.0 Ethernet controller: Digium, Inc. Wildcard TDM410 4-port analog
> card (rev 11)
>
>          Subsystem: Digium, Inc. Wildcard TDM410 4-port analog card
>
>          Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
>          Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
>  >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>
>          Interrupt: pin A routed to IRQ 16
>
>          Region 0: I/O ports at 5000 [disabled] [size=256]
>
>          Region 1: Memory at fdef0000 (32-bit, non-prefetchable)
> [disabled] [size=1K]
>
>          [virtual] Expansion ROM at f0000000 [disabled] [size=128K]
>
>          Capabilities: [c0] Power Management version 2
>
>                  Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=100mA
> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>
>                  Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
>
>          Kernel driver in use: pciback
>
> Guest DomU (Centos 6.4):
>
> kernel-xen-3.9.2-1.el6xen.x86_64
>
> Created using virt-install onto a 20G LVM with 1024Mb ram
>
> XML for DomU dumped and converted to native then the domain destroyed
> and undefined and recreated using XL create with the new cfg file.  This
> is to allow inclusion of pci [‘06:01.0’] parameter in config.
>
> Using the static setup, I can get Dom0 to hide the PCI device.  I can
> also achieve the same effect with the dynamic set up using
> pci-assignable-attach and pci-attach. Here is the dmesg relating to the
> device. Reg 30 is highlighted because this seems to be where the problem is.
>
> pci 0000:06:01.0: [d161:8005] type 00 class 0x020000
>
> pci 0000:06:01.0: reg 10: [io  0x5000-0x50ff]
>
> pci 0000:06:01.0: reg 14: [mem 0xfdef0000-0xfdef03ff]
>
> pci 0000:06:01.0: *reg 30: [mem 0x00000000-0x0001ffff pref]*
>
> pci 0000:06:01.0: supports D1 D2
>
> pci 0000:06:01.0: PME# supported from D0 D1 D2 D3hot D3cold
>
> pci 0000:06:01.0: BAR 6: assigned [mem 0xf0000000-0xf001ffff pref]
>
> pciback 0000:06:01.0: seizing device
>
> pciback 0000:06:01.0: PCI IRQ 48 -> rerouted to legacy IRQ 16
>
> pciback 0000:06:01.0: PCI IRQ 48 -> rerouted to legacy IRQ 16
>
> xen-pciback: vpci: 0000:06:01.0: assign to virtual slot 0
>
> In the Dom0 I can define the device statically in the config file or
> dynamically as described above.  Both scenarios result in the same error
> being displayed.
>
> pcifront pci-0: Installing PCI frontend
>
> pcifront pci-0: Creating PCI Frontend Bus 0000:00
>
> pcifront pci-0: PCI host bridge to bus 0000:00
>
> pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
>
> pci_bus 0000:00: root bus resource [mem 0x00000000-0xfffffffff]
>
> pci_bus 0000:00: root bus resource [bus 00-ff]
>
> pci 0000:00:00.0: [d161:8005] type 00 class 0x020000
>
> pci 0000:00:00.0: reg 10: [io  0x5000-0x50ff]
>
> pci 0000:00:00.0: reg 14: [mem 0xfdef0000-0xfdef03ff]
>
> pci 0000:00:00.0: *reg 30: [mem 0xf0000000-0xffffffff pref]*
>
> pci 0000:00:00.0: supports D1 D2
>
> pcifront pci-0: claiming resource 0000:00:00.0/0
>
> pcifront pci-0: claiming resource 0000:00:00.0/1
>
> pcifront pci-0: claiming resource 0000:00:00.0/6
>
> pci 0000:00:00.0: address space collision: [mem 0xf0000000-0xffffffff
> pref] conflicts with 0000:00:00.0 [mem 0xfdef0000-0xfdef03ff]
>
> pcifront pci-0: Could not claim resource 0000:00:00.0/6! Device offline.
> Try using e820_host=1 in the guest config.
>
> This appears to show that the PCI device is conflicting with itself (reg
> 14 with reg 30) because the address space for reg 30 is different in
> pciback to pcifront.
>
> I have tried setting up the domain with both XM and XL with the same result
>
> Adding passthrough and permissive settings with no change
>
> Adding iommu=soft to guest kernel command line.
>
> I’ve tried adding the e820_host flag to the config file but this doesn’t
> seem to solve anything.
>
> Different Xen enabled kernels.
>
> Wiping the server and rebuilding the whole thing from scratch (more than
> once)
>
> The Digium PCI card works fine on a normal Centos 6.3 setup with no Xen.
>
> I’m out of ideas now on how to solve this, so if anyone has made this
> card work by doing something different, I’d be grateful for  any
> suggestions.  I’ve looked at the source for pcifont.c and come to the
> conclusion that my c coding skills are not going to be good enough to
> debug/change this program.
>
> I can provide more dmesg outputs or other documentation if needed.
>
> Thanks in advance for any help
>
> Jon
>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxx
> http://lists.xen.org/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users