We’re running 64-bit Xex 4.1.1 and 32-bit Linux 3.0.4 Dom0 (Linux 3.1 shows the same symptom.)
Several PCI drivers are unable to use DMA. Most fallback to using PIO but in two instances the network drivers (e1000 and pcinet32) abort. The same kernel running on the same hardware without Xen works fine.
Digging through the code, in swiotlb-xen.c I find “DMA_BIT_MASK(32)” (0x00000000ffffffff) compared to “xen_virt_to_bus(xen_io_tlb_end - 1)” which resolves to 0x1,20fd,feff. Since the address is larger than the mask, DMA is declared as unsupportable.
In talking with others I hear Linux handles this situation with bounce buffers. Is there a config setting I’ve missed to enable that for Xen? (Config file attached)
Relevant slice of source callback list:
xen_swiotlb_dma_supported (drivers/xen/swiotlb-xen.c: line 591)
dma_supported (arch/x86/kernel/pci-dma.c: line 199)
dma_set_mask (arch/x86/kernel/pci-dma.c: line 59)
e1000_probe (drivers/net/e1000/e1000_main.c: line 986)
Relevant patches:
https://lkml.org/lkml/2011/9/1/100, "[PATCH v2] xen: x86_32: do not enable iterrupts when returning from exception in interrupt context"
http://xen.1045712.n5.nabble.com/PATCH-mm-sync-vmalloc-address-space-page-tables-in-alloc-vm-area-td4757995.html
"[PATCH] mm: sync vmalloc address space page tables in alloc_vm_area()" (this patch was reverted for 3.1 but this is 3.0.4) and
an additional 2048 NR_IRQS to support (as I understand it) all the virtual devices we might support with 50 guests.
Not so relevant patches in md, nbd and loop.
lspci:
00:00.0 Host bridge: Intel Corporation E7520 Memory Controller Hub (rev 09)
00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express Port A (rev 09)
00:04.0 PCI bridge: Intel Corporation E7525/E7520 PCI Express Port B (rev 09)
00:05.0 PCI bridge: Intel Corporation E7520 PCI Express Port B1 (rev 09)
00:06.0 PCI bridge: Intel Corporation E7520 PCI Express Port C (rev 09)
00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #3 (rev 02)
00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2)
00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC Interface Bridge (rev 02)
00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE Controller (rev 02)
01:00.0 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor (A-Segment Bridge) (rev 06)
01:00.2 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor (B-Segment Bridge) (rev 06)
02:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 08)
05:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge A (rev 09)
05:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge B (rev 09)
06:07.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller (rev 05)
07:08.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller (rev 05)
09:05.0 Class ff00: Dell Remote Access Card 4 Daughter Card
09:05.1 Class ff00: Dell Remote Access Card 4 Daughter Card Virtual UART
09:05.2 Class ff00: Dell Remote Access Card 4 Daughter Card SMIC interface
09:06.0 IDE interface: Silicon Image, Inc. PCI0680 Ultra ATA-133 Host Controller (rev 02)
09:0d.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY [Radeon 7000/VE]
1 00:00.0 Host bridge: Intel Corporation E7520 Memory Controller Hub (rev 09)
2 00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express Port A (rev 09)
3 00:04.0 PCI bridge: Intel Corporation E7525/E7520 PCI Express Port B (rev 09)
4 00:05.0 PCI bridge: Intel Corporation E7520 PCI Express Port B1 (rev 09)
5 00:06.0 PCI bridge: Intel Corporation E7520 PCI Express Port C (rev 09)
6 00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #1 (rev 02)
7 00:1d.1 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #2 (rev 02)
8 00:1d.2 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #3 (rev 02)
9 00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02)
10 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2)
11 00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC Interface Bridge (rev 02)
12 00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE Controller (rev 02)
13 01:00.0 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor (A-Segment Bridge) (rev 06)
14 01:00.2 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor (B-Segment Bridge) (rev 06)
15 02:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 08)
16 05:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge A (rev 09)
17 05:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge B (rev 09)
18 06:07.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller (rev 05)
19 07:08.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller (rev 05)
20 09:05.0 Class ff00: Dell Remote Access Card 4 Daughter Card
21 09:05.1 Class ff00: Dell Remote Access Card 4 Daughter Card Virtual UART
22 09:05.2 Class ff00: Dell Remote Access Card 4 Daughter Card SMIC interface
23 09:06.0 IDE interface: Silicon Image, Inc. PCI0680 Ultra ATA-133 Host Controller (rev 02)
24 09:0d.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY [Radeon 7000/VE]
06:07.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller (rev 05)
Subsystem: Dell PRO/1000 MT Network Connection
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Interrupt: pin A routed to IRQ 64
Region 0: Memory at dfae0000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at dcc0 [size=64]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [e4] PCI-X non-bridge device
Command: DPERE- ERO+ RBC=512 OST=1
Status: Dev=00:00.0 64bit- 133MHz- SCD- USC- DC=simple DMMRBC=2048 DMOST=1 DMCRS=8 RSCEM- 266MHz- 533MHz-
log file attached.
