[Xen-users] VT-d passthrough - NMI crashes system

[Thomas Prause <tprause@xxxxxxxx>]

I want to pass some FC adapters to the domU following the steps
described in the VTdHowTo -> http://wiki.xensource.com/xenwiki/VTdHowTo
using XEN 3.3.1 (SLES 11)

dom0 seems to be configured properly:
# xm pci-list-assignable-devices
0000:15:00.0
0000:24:00.0
0000:1f:00.0
0000:1a:00.0
#

But after starting a domU with one adapter assigned it does not show up
to lspci in the domU. Even 'xm pci-list-assignable-devices' does not
show this one adapter any longer.

After some seconds or minutes the whole system crashes, tries to reboot,
fails with NMI errors in the BIOS log and finally stops in the BIOS with
an error that 3 consecutive boot failures have occured and now the
default settings will be used :-o

System can then be booted and the log of the dom0 shows:

Mar  2 12:52:27 manni logger: /etc/xen/scripts/block: Writing
backend/vbd/1/5632/hotplug-status connected to xenstore.
Mar  2 12:52:27 manni kernel: vbd vbd-1-5632: 2 creating vbd structure
Mar  2 12:52:36 manni kernel: vif1.0: no IPv6 routers present
Mar  2 12:52:41 manni kernel: blkback: ring-ref 8, event-channel 8,
protocol 1 (x86_64-abi)
Mar  2 12:52:41 manni kernel: blkback: ring-ref 9, event-channel 9,
protocol 1 (x86_64-abi)
Mar  2 12:52:46 manni kernel: Uhhuh. NMI received for unknown reason 00.
Mar  2 12:52:46 manni kernel: Do you have a strange power saving mode
enabled?
Mar  2 12:52:46 manni kernel: Dazed and confused, but trying to continue
Mar  2 12:53:33 manni sshd[7298]: Accepted keyboard-interactive/pam for
root from xx.xx.xx.xx port 42896 ssh2
Mar  3 16:17:26 manni syslog-ng[3708]: syslog-ng starting up;
version='2.0.9'

I assumed some sort of IRQ conflict and checked:
# dmesg | grep GSI
IOAPIC[0]: apic_id 8, version 0, address 0xfec00000, GSI 0-23
IOAPIC[1]: apic_id 9, version 0, address 0xfec80000, GSI 24-47
pci 0000:00:01.0: PCI INT A -> GSI 28 (level, low) -> IRQ 28
pci 0000:00:02.0: PCI INT A -> GSI 29 (level, low) -> IRQ 29
pci 0000:00:03.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
pci 0000:00:05.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26
pci 0000:00:07.0: PCI INT A -> GSI 30 (level, low) -> IRQ 30
pci 0000:00:09.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
pci 0000:00:1c.4: PCI INT A -> GSI 16 (level, low) -> IRQ 16
pci 0000:06:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ata_piix 0000:00:1f.2: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ata_piix 0000:00:1f.5: PCI INT C -> GSI 21 (level, low) -> IRQ 21
megaraid_sas 0000:01:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
pciback 0000:24:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
pciback 0000:1f:00.0: PCI INT A -> GSI 30 (level, low) -> IRQ 30
pciback 0000:1a:00.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26
pciback 0000:15:00.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
 sda:<6>ehci_hcd 0000:00:1a.7: PCI INT C -> GSI 19 (level, low) -> IRQ 19
ehci_hcd 0000:00:1d.7: PCI INT A -> GSI 17 (level, low) -> IRQ 17
uhci_hcd 0000:00:1a.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
uhci_hcd 0000:00:1a.1: PCI INT B -> GSI 18 (level, low) -> IRQ 18
uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 18 (level, low) -> IRQ 18
uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 19 (level, low) -> IRQ 19
bnx2 0000:0b:00.0: PCI INT A -> GSI 28 (level, low) -> IRQ 28
bnx2 0000:0b:00.1: PCI INT B -> GSI 40 (level, low) -> IRQ 40
i801_smbus 0000:00:1f.3: PCI INT B -> GSI 22 (level, low) -> IRQ 22
#

The IRQs for the FC adapters are used by other devices. These are PCI
bridges:
# lspci
00:00.0 Host bridge: Intel Corporation X58 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
1 (rev 13)
00:02.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
2 (rev 13)
00:03.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
3 (rev 13)
00:05.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
5 (rev 13)
00:07.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
7 (rev 13)
00:09.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
9 (rev 13)

So I tried to passthrough the corresponding PCI with the FC adapter. But
this failed as well:
 # cat /sys/bus/pci/drivers/pciback/slots
0000:15:00.0
0000:1a:00.0
0000:1f:00.0
0000:24:00.0
0000:00:03.0
 # xm create /etc/xen/vm/sid-sles11 pci=00:00:03.0 pci=00:15:00.0
Using config file "/etc/xen/vm/sid-sles11".
Error: pci: PCI Backend does not own device 0000:00:03.0
See the pciback.hide kernel command-line parameter or
bind your slot/device to the PCI backend using sysfs
 #

Am I on the wrong track with the PCI bridge? Or is this some sort of
hardware issue (IBM X3650 M2 with latest firmware)?

Any thoughts are highly appreciated.
Thanks.

Thomas

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users