Re: [Xen-users] Problem with PCI Pass-through address space collision

[Jon Skilling <jon_skilling@xxxxxxxxxxx>]

Hi Gordan,

Thanks for your reply.  It is my understanding that PCI pass-through should
work for PV guests, which is what I am creating here.  One thing that I
notice though, other than XL DMESG giving the message about Vt-d being
disabled, is that other than the command line message, iommu=soft doesn't
seem to produce any confirmation messages anywhere so I can't really tell if
this is running or not.

I have to confess that I have not seen any documentation on XL stating that
the device must be detached from Dom0 before it can be attached to a guest.
Certainly, XL pci-detach freePBX 06:01.0 produces an error saying that the
domain is invalid.  However, I tried your suggestion prior to starting the
domain and it runs without complaint and says that the device has been
detached.

I reincorporated your suggestion into xen-pciback.conf and added modprobe
xen-pciback into rc.local to get everything to start at boot up and
restarted the machine.
I then ran the virsh-nodev-detach command followed by xl create -c
freePBX.cfg.  Unfortunately, this produced the same result as before.

I've also tried adding iommu=soft to the Dom0 kernel line, but as I said,
this didn't produce any discernible differences to the XL DMESG output.

Instead of hotplugging, I tried adding the two last lines to the end of the
freePBX.cfg file:

name = "freePBX"
uuid = "91cd5696-1451-c333-f6a4-1628a79976bd"
maxmem = 1024
memory = 1024
vcpus = 1
bootloader = "pygrub"
localtime = 0
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
disk = [ "phy:/dev/VolGroup/freePBX,xvda,w" ]
vif = [ "mac=00:16:3e:00:9a:9c,bridge=br0,script=vif-bridge,vifname=vif2.0"
]

extra = "iommu=soft debug loglevel=10 earlyprintk=xenboot console=hvc0 ro
xencons=tty"
pci = ['06:01.0,permissive=1']

As you can see from above, I'm allocating 1024Mb to the guest.  I haven't
got time now until the weekend to apply the 4.2.2-5 patches to check if they
solve my problem, but I'll post a message when I've done it to update on how
it went.  Suffice to say, I'm still getting the same error message :-(

Thanks for your help and input.

Jon

-----Original Message-----
From: Gordan Bobic [mailto:gordan@xxxxxxxxxx] 
Sent: 21 May 2013 07:36
To: Jon Skilling
Cc: xen-users@xxxxxxxxxxxxx
Subject: Re: [Xen-users] Problem with PCI Pass-through address space
collision

I'm pretty sure I seem to recall that PCI passthrough will not work without
VT-d, but by all means, feel free to try.

Even if you did have working VT-d, though, you have to detach the device
from dom0 before you can add it to domU, using something like:

virsh nodedev-detach pci_0000_06_01_0

Given the EL6 CRC Xen packages you are using, they use pciback built as a
module, so kernel boot parameters won't help. What you need to do is add
this to /etc/modprobe.d/:

# cat xen-pciback.conf
options xen-pciback permissive=1 hide=(06:01.0)

Run depmod -a once you have done that.

Then:

# modprobe xen-pciback
virsh nodedev-detach pci_0000_06_01_0

Also add the driver for the card to /etc/modprobe.d/blacklist.conf.

After that you should be able to boot the domU with the device.

You may also want to upgrade to the latest testing packages (4.2.2-5) since
they include a PCI passthrough fix from a couple of days ago, although it
doesn't look like you are falling foul of it.

Also, how much RAM are you passing to domU? Try giving it <= 2GB. There is a
PCI memory map bug that can cause a nasty memory stomp that kept me chasing
my tail for days. For most people it manifests at > 4GB, but on my system it
manifested at > 2GB.

HTH.

Gordan

On 05/20/2013 06:04 PM, Jon Skilling wrote:
> Hi,
>
> I've been trying to configure Xeon on my HP ML350 G4 server for the 
> past two weeks and despite reading just about every word of the Xen 
> wiki and numerous other posts and mails, I can't find a solution to my
problem.
> Any help on this would be much appreciated!
>
> Setup:
>
> HP ML350 G4, Dual xeon, 6Gb Ram, 6 disk scsi raid array,  Digium 
> TDM410P analogue PBX card on PCI.  Hardware virtualization (Vt-d) is 
> not an option with this machine.
>
> I followed these instructions (more or less) to set up Dom0 and DomU:
>
> http://www.howtoforge.com/virtualization-with-xen-on-centos-6.3-x86_64
> -paravirtualization-and-hardware-virtualization
>
> with the following changes:
>
> Host Dom0 (Centos 6.4):
>
> xen-4.2.2-4.el6.x86_64
>
> kernel-xen-3.9.2-1.el6xen.x86_64
>
> libvirt 1.0.3-1 (python-virtinstall causes libvirt to be upgraded to 
> 1.0.3. From checking the source, the Xen patch appears to be there 
> already, so no recompile needed - the Xen patch doesn't work with this 
> source anyway.
>
> XEND has been disabled from boot up because it causes problems with XL 
> tools although the same address space collision occurs if I use the XM 
> tool set.
>
> I have tried xen-pciback.hide(06:01.0)  on the kernel module 
> definitions in boot.conf but this doesn't seem to do anything. Adding 
> records to modprobe.conf and rc.local work better.
>
> The device I'm trying to passthrough is defined:
>
> 06:01.0 Ethernet controller: Digium, Inc. Wildcard TDM410 4-port 
> analog card (rev 11)
>
>          Subsystem: Digium, Inc. Wildcard TDM410 4-port analog card
>
>          Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>
>          Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
>  >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>
>          Interrupt: pin A routed to IRQ 16
>
>          Region 0: I/O ports at 5000 [disabled] [size=256]
>
>          Region 1: Memory at fdef0000 (32-bit, non-prefetchable) 
> [disabled] [size=1K]
>
>          [virtual] Expansion ROM at f0000000 [disabled] [size=128K]
>
>          Capabilities: [c0] Power Management version 2
>
>                  Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=100mA
> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>
>                  Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 
> PME-
>
>          Kernel driver in use: pciback
>
> Guest DomU (Centos 6.4):
>
> kernel-xen-3.9.2-1.el6xen.x86_64
>
> Created using virt-install onto a 20G LVM with 1024Mb ram
>
> XML for DomU dumped and converted to native then the domain destroyed 
> and undefined and recreated using XL create with the new cfg file.  
> This is to allow inclusion of pci ['06:01.0'] parameter in config.
>
> Using the static setup, I can get Dom0 to hide the PCI device.  I can 
> also achieve the same effect with the dynamic set up using 
> pci-assignable-attach and pci-attach. Here is the dmesg relating to 
> the device. Reg 30 is highlighted because this seems to be where the
problem is.
>
> pci 0000:06:01.0: [d161:8005] type 00 class 0x020000
>
> pci 0000:06:01.0: reg 10: [io  0x5000-0x50ff]
>
> pci 0000:06:01.0: reg 14: [mem 0xfdef0000-0xfdef03ff]
>
> pci 0000:06:01.0: *reg 30: [mem 0x00000000-0x0001ffff pref]*
>
> pci 0000:06:01.0: supports D1 D2
>
> pci 0000:06:01.0: PME# supported from D0 D1 D2 D3hot D3cold
>
> pci 0000:06:01.0: BAR 6: assigned [mem 0xf0000000-0xf001ffff pref]
>
> pciback 0000:06:01.0: seizing device
>
> pciback 0000:06:01.0: PCI IRQ 48 -> rerouted to legacy IRQ 16
>
> pciback 0000:06:01.0: PCI IRQ 48 -> rerouted to legacy IRQ 16
>
> xen-pciback: vpci: 0000:06:01.0: assign to virtual slot 0
>
> In the Dom0 I can define the device statically in the config file or 
> dynamically as described above.  Both scenarios result in the same 
> error being displayed.
>
> pcifront pci-0: Installing PCI frontend
>
> pcifront pci-0: Creating PCI Frontend Bus 0000:00
>
> pcifront pci-0: PCI host bridge to bus 0000:00
>
> pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
>
> pci_bus 0000:00: root bus resource [mem 0x00000000-0xfffffffff]
>
> pci_bus 0000:00: root bus resource [bus 00-ff]
>
> pci 0000:00:00.0: [d161:8005] type 00 class 0x020000
>
> pci 0000:00:00.0: reg 10: [io  0x5000-0x50ff]
>
> pci 0000:00:00.0: reg 14: [mem 0xfdef0000-0xfdef03ff]
>
> pci 0000:00:00.0: *reg 30: [mem 0xf0000000-0xffffffff pref]*
>
> pci 0000:00:00.0: supports D1 D2
>
> pcifront pci-0: claiming resource 0000:00:00.0/0
>
> pcifront pci-0: claiming resource 0000:00:00.0/1
>
> pcifront pci-0: claiming resource 0000:00:00.0/6
>
> pci 0000:00:00.0: address space collision: [mem 0xf0000000-0xffffffff 
> pref] conflicts with 0000:00:00.0 [mem 0xfdef0000-0xfdef03ff]
>
> pcifront pci-0: Could not claim resource 0000:00:00.0/6! Device offline.
> Try using e820_host=1 in the guest config.
>
> This appears to show that the PCI device is conflicting with itself 
> (reg
> 14 with reg 30) because the address space for reg 30 is different in 
> pciback to pcifront.
>
> I have tried setting up the domain with both XM and XL with the same 
> result
>
> Adding passthrough and permissive settings with no change
>
> Adding iommu=soft to guest kernel command line.
>
> I've tried adding the e820_host flag to the config file but this 
> doesn't seem to solve anything.
>
> Different Xen enabled kernels.
>
> Wiping the server and rebuilding the whole thing from scratch (more 
> than
> once)
>
> The Digium PCI card works fine on a normal Centos 6.3 setup with no Xen.
>
> I'm out of ideas now on how to solve this, so if anyone has made this 
> card work by doing something different, I'd be grateful for  any 
> suggestions.  I've looked at the source for pcifont.c and come to the 
> conclusion that my c coding skills are not going to be good enough to 
> debug/change this program.
>
> I can provide more dmesg outputs or other documentation if needed.
>
> Thanks in advance for any help
>
> Jon
>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxx
> http://lists.xen.org/xen-users
>



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users