[Xen-users] Re: Unable to pass device to IOMMU

[Adam Moss <amoss@xxxxxxxxxx>]

Hi all,

I have worked with Michael on this problem, here are some further details that may help troubleshoot the issue:

The error seen when starting DomU is in fact "Failed to assign device to IOMMU".

We're using the following system setup for our Xen environment:

SunBlade 6000 chassis
SunBlade X6270 blade
SuSE ES11 SP1 64bit Dom0 (Xen 4.0.0 - Kernel 2.6.32.12-0.7-xen )
SuSE ES11 SP1 64bit DomU

The lspci -vt output at Dom0 looks like this:

###
-[0000:00]-+-00.0Â Intel Corporation 5520 I/O Hub to ESI Port
ÂÂÂÂÂÂÂÂÂÂ +-01.0-[0000:01-06]----00.0Â LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS
ÂÂÂÂÂÂÂÂÂÂ +-03.0-[0000:07-0c]--
ÂÂÂÂÂÂÂÂÂÂ +-05.0-[0000:0d-12]----00.0-[0000:0e]----00.0Â Freescale Semiconductor Inc Device 0091
ÂÂÂÂÂÂÂÂÂÂ +-07.0-[0000:13-18]--
ÂÂÂÂÂÂÂÂÂÂ +-09.0-[0000:19-1e]--
ÂÂÂÂÂÂÂÂÂÂ +-13.0Â Intel Corporation 5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller
ÂÂÂÂÂÂÂÂÂÂ +-14.0Â Intel Corporation 5520/5500/X58 I/O Hub System Management Registers
ÂÂÂÂÂÂÂÂÂÂ +-14.1Â Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers
ÂÂÂÂÂÂÂÂÂÂ +-14.2Â Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers
ÂÂÂÂÂÂÂÂÂÂ +-14.3Â Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers
ÂÂÂÂÂÂÂÂÂÂ +-16.0Â Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device
ÂÂÂÂÂÂÂÂÂÂ +-16.1Â Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device
ÂÂÂÂÂÂÂÂÂÂ +-16.2Â Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device
ÂÂÂÂÂÂÂÂÂÂ +-16.3Â Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device
ÂÂÂÂÂÂÂÂÂÂ +-16.4Â Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device
ÂÂÂÂÂÂÂÂÂÂ +-16.5Â Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device
ÂÂÂÂÂÂÂÂÂÂ +-16.6Â Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device
ÂÂÂÂÂÂÂÂÂÂ +-16.7Â Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device
ÂÂÂÂÂÂÂÂÂÂ +-1a.0Â Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
ÂÂÂÂÂÂÂÂÂÂ +-1a.1Â Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
ÂÂÂÂÂÂÂÂÂÂ +-1a.2Â Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
ÂÂÂÂÂÂÂÂÂÂ +-1a.7Â Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
ÂÂÂÂÂÂÂÂÂÂ +-1c.0-[0000:1f]--+-00.0Â Intel Corporation 82575EB Gigabit Network Connection
ÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ \-00.1Â Intel Corporation 82575EB Gigabit Network Connection
ÂÂÂÂÂÂÂÂÂÂ +-1d.0Â Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
ÂÂÂÂÂÂÂÂÂÂ +-1d.1Â Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
ÂÂÂÂÂÂÂÂÂÂ +-1d.2Â Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
ÂÂÂÂÂÂÂÂÂÂ +-1d.7Â Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
ÂÂÂÂÂÂÂÂÂÂ +-1e.0-[0000:20]----05.0Â ASPEED Technology, Inc. ASPEED Graphics Family
ÂÂÂÂÂÂÂÂÂÂ +-1f.0Â Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
ÂÂÂÂÂÂÂÂÂÂ +-1f.2Â Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
ÂÂÂÂÂÂÂÂÂÂ \-1f.3Â Intel Corporation 82801JI (ICH10 Family) SMBus Controller
###

The following commands are used to hide the single device (0000:0e:00.0) we want to pass back to DomU:

###
modprobe pciback

echo -n "0000:0e:00.0" > /sys/bus/pci/drivers/pciback/new_slot
echo -n "0000:0e:00.0" > /sys/bus/pci/drivers/pciback/bind
###

After this the device is shown correctly in the assignable devices list.

###
SunBlade2:/etc/xen/vm # xm pci-list-assignable-devices
0000:0e:00.0
###

The DomU is then started/created using the following configuration file:

###
import os, re
arch = os.uname()[4]
if re.search('64', arch):
ÂÂÂ arch_libdir = 'lib64'
else:
ÂÂÂ arch_libdir = 'lib'
kernelÂÂÂÂÂÂÂÂÂ = "/usr/lib/xen/boot/hvmloader"
builderÂÂÂÂÂÂÂÂ = 'hvm'
device_modelÂÂÂ = '/usr/' + arch_libdir + '/xen/bin/qemu-dm'
nameÂÂÂÂÂÂÂÂÂÂÂ = "node1"
vcpusÂÂÂÂÂÂÂÂÂÂ =Â 1
memoryÂÂÂÂÂÂÂÂÂ =Â 1024
paeÂÂÂÂÂÂÂÂÂÂÂÂ =Â 1
acpiÂÂÂÂÂÂÂÂÂÂÂ =Â 1
apicÂÂÂÂÂÂÂÂÂÂÂ =Â 1

bootÂÂÂÂÂÂÂÂÂÂÂ = "dc"
vifÂÂÂÂÂÂÂÂÂÂÂÂ =['mac=00:11:22:33:00:04, bridge=br0, model=e1000']
diskÂÂÂÂÂÂÂÂÂÂÂ =['phy:sda5,hda,w']

on_poweroffÂÂÂÂ = 'destroy'
on_rebootÂÂÂÂÂÂ = 'restart'
on_crashÂÂÂÂÂÂÂ = 'restart'

snapshotÂÂÂÂÂÂÂ =Â 0
sdlÂÂÂÂÂÂÂÂÂÂÂÂ =Â 0
openglÂÂÂÂÂÂÂÂÂ =Â 1
vncÂÂÂÂÂÂÂÂÂÂÂÂ =Â 1
vnclistenÂÂÂÂÂÂ = "127.0.0.1"
vncdisplayÂÂÂÂÂ =Â 4
vncunusedÂÂÂÂÂÂ =Â 0
vncpasswdÂÂÂÂÂÂ = ''
nographicÂÂÂÂÂÂ =Â 0
stdvgaÂÂÂÂÂÂÂÂÂ =Â 0
serialÂÂÂÂÂÂÂÂÂ = 'pty'

pciÂÂÂÂÂÂÂÂÂÂ = ['0000:0e:00.0']
###

After running the create command the "failed to assign to IOMMU" error is seen:

###
SunBlade2:/etc/xen/vm # xm create node1.conf
Using config file "./node1.conf".
Error: Failed to assign device to IOMMU (0000:0e:00.0@100,msitranslate=1,power_mgmt=0)
###

We have enabled the debug traces (iommu=verbose loglvl=all guest_loglvl=all) which show the following at the point of failure (Please ignore the time stamps):

/var/log/messages
--------------------------

Mar 10 02:57:05 SunBlade2 kernel: [ 1495.751322] (cdrom_add_media_watch() file=/usr/src/packages/BUILD/kernel-xen-2.6.32.12/linux-2.6.32/drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/2/768
Mar 10 02:57:05 SunBlade2 kernel: [ 1495.751327] (cdrom_is_type() file=/usr/src/packages/BUILD/kernel-xen-2.6.32.12/linux-2.6.32/drivers/xen/blkback/cdrom.c, line=95) type:0
Mar 10 02:57:05 SunBlade2 kernel: [ 1495.936015] pcieport 0000:00:05.0: AER: Uncorrected (Non-Fatal) error received: id=0028
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.023732] pcieport 0000:00:05.0: PCIE Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0028(Requester ID)
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.023737] pcieport 0000:00:05.0:ÂÂ device [8086:340c] error status/mask=00004000/00000000
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.023742] pcieport 0000:00:05.0:ÂÂÂ [14] Completion TimeoutÂÂÂÂ (First)
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024193] pcieport 0000:00:05.0: broadcast error_detected message
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024198] pciback 0000:0e:00.0: pciback device is not found/assigned
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024201] pcieport 0000:00:05.0: broadcast mmio_enabled message
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024203] pciback 0000:0e:00.0: pciback device is not found/assigned
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024205] pcieport 0000:00:05.0: broadcast resume message
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024208] pciback 0000:0e:00.0: pciback device is not found/assigned
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024213] pcieport 0000:00:05.0: AER driver successfully recovered
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024216] pcieport 0000:00:05.0: AER: Uncorrected (Non-Fatal) error received: id=0028
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024305] pcieport 0000:00:05.0: can't find device of ID0028
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024307] pcieport 0000:00:05.0: AER: Uncorrected (Non-Fatal) error received: id=0028
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024395] pcieport 0000:00:05.0: can't find device of ID0028
Mar 10 02:57:05 SunBlade2 kernel: [ 1495.587191] device tap2.0 entered promiscuous mode
Mar 10 02:57:05 SunBlade2 kernel: [ 1495.587205] br0: port 3(tap2.0) entering forwarding state
Mar 10 02:57:05 SunBlade2 kernel: [ 1495.771897] br0: port 3(tap2.0) entering disabled state
Mar 10 02:57:06 SunBlade2 kernel: [ 1495.959814] device tap2.0 left promiscuous mode

/var/log/xen/xend.log
----------------------------

[2011-03-10 02:58:42 5613] DEBUG (XendDomainInfo:825) XendDomainInfo.hvm_pci_device_insert_dev: 0000:0e:00.0@100,msitranslate=1,power_mgmt=0
[2011-03-10 02:58:42 5613] ERROR (XendDomainInfo:2957) XendDomainInfo.initDomain: exception occurred
Traceback (most recent call last):
File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 2949, in _initDomain
ÂÂÂ self._createDevices()
 File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 2440, in _createDevices
ÂÂÂ self.pci_device_configure_boot()
 File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 641, in pci_device_configure_boot
ÂÂÂ self.pci_device_configure(dev_sxp, first_dev = first)
 File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 934, in pci_device_configure
ÂÂÂ vdevfn = self.hvm_pci_device_insert(dev_config)
 File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 800, in hvm_pci_device_insert
ÂÂÂ return self.hvm_pci_device_insert_dev(new_dev)
 File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 828, in hvm_pci_device_insert_dev
ÂÂÂ raise VmError("Failed to assign device to IOMMU (%s)" % bdf_str)
VmError: Failed to assign device to IOMMU (0000:0e:00.0@100,msitranslate=1,power_mgmt=0)
[2011-03-10 02:58:42 5613] ERROR (XendDomainInfo:486) VM start failed
Traceback (most recent call last):
 File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 472, in start
ÂÂÂ XendTask.log_progress(31, 60, self._initDomain)
 File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py", line 209, in log_progress
ÂÂÂ retval = func(*args, **kwds)
 File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 2960, in _initDomain
ÂÂÂ raise exn
VmError: Failed to assign device to IOMMU (0000:0e:00.0@100,msitranslate=1,power_mgmt=0)


xm dmes
-------------

(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 184kB init memory.
(XEN) mtrr: type mismatch for fb000000,800000 old: write-back new: write-combining
(XEN) [VT-D]iommu.c:1461: d0:PCI: unmap bdf = e:0.0
(XEN) [VT-D]iommu.c:1332: d1:PCI: map bdf = e:0.0
(XEN) domctl.c:853:d0 XEN_DOMCTL_assign_device: assign device (e:0.0) failed
(XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 1, error -22
(XEN) [VT-D]iommu.c:1461: d0:PCI: unmap bdf = e:0.0
(XEN) [VT-D]iommu.c:1332: d2:PCI: map bdf = e:0.0
(XEN) domctl.c:853:d0 XEN_DOMCTL_assign_device: assign device (e:0.0) failed
(XEN) [VT-D]iommu.c:1461: d0:PCI: unmap bdf = e:0.0
(XEN) [VT-D]iommu.c:1332: d3:PCI: map bdf = e:0.0
(XEN) domctl.c:853:d0 XEN_DOMCTL_assign_device: assign device (e:0.0) failed
(XEN) [VT-D]iommu.c:1461: d0:PCI: unmap bdf = e:0.0
(XEN) [VT-D]iommu.c:1332: d4:PCI: map bdf = e:0.0
(XEN) domctl.c:853:d0 XEN_DOMCTL_assign_device: assign device (e:0.0) failed

The full trace (xm dmes, xend.log) output and other relevant information from the system is attached.

Things to note
===========

1) The problem still occurs if we try and hotplug the device using "xm pci-attach ..."
2) If the same hardware and software configuration is used with Xen 3.3.1 at Dom0 (SLES11) the problem is not seen.
3) We have compiled Xen 4.0.1 for SLES11 SP1 at Dom0 and still see the same problem.
4) We have tried passing back the PLX bridge device "0d:00.0" associated with the device as well as the device itself however we then see the error "Error: pci: PCI Backend and pci-stub don't own device 0000:0d:00.0" during DomU startup even though 0d:00.0 is shown in the assignable device list.

Looking at recent posts this problems looks very similar to the one in subject "PCI Passthrough Problems/Questions" posted today.

Any pointers at this stage would be great.
Thanks,
Adam

----------
Adam Moss

Adax Europe Ltd
Email : Adam.Moss@xxxxxxxxxx

On 21/09/2010 20:05, Michael Maher wrote:

Hello,

I was wondering if anybody had any insight they could lend us on this?

Your help would be very much appreciated,


Many Thanks,
Michael




On Thu, 2010-09-16 at 18:44 +0100, Michael Maher wrote:

Hello,

 

Iâm hoping someone may be able to push us in the right direction. Iâm
trying to get one of our products to work with the current branch of
Xen (4.0/4.0.0) but Iâm hitting a problem.

 

We are currently using Xen 3.3.2 on a Sun Blade 600 chassis (with
x6270 blades) this works great.

 

But when we upgrade to Xen 4.0/4.0.1 we get the following message when
starting the DomU:

                Device Cannot be passed to IOMMU

 

I understand from the release note that there were some changes made
to the IOMMU/pciback pass-through in this release, does anyone have
any details as to what might be going wrong?

 

Weâre using SLES11/sp1 and Iâve tried with the Novell build and
compiled it myself. I also have the latest firmware for the chassis
and blade from Sun, all of which show the same problem.

 

Has anyone seen to same thing?

Any help would be greatly appreciated,

 

 

Many Thanks,

Michael


Michael Maher

Adax Europe Ltd

mmaher@xxxxxxxxxx

Attachment: Xen-Issue-Logs.txt
Description: Text document

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users