[Xen-devel] Xen, MCFG acpi table and E820 address map

[Santosh Jodh <Santosh.Jodh@xxxxxxxxxx>]

Xen will use information from MCFG acpi table to access PCIe extended 
configuration space. However, Xen validates MCFG table by making sure that the 
addresses specified in the MCFG table is correctly marked as reserved in the 
E820 address map. If it is not, the MCFG table is ignored - thereby preventing 
Xen from accessing PCIe extended configuration space.

I recently came across a workstation class system that supports VT-d. This 
system BIOS has a valid MCFG table. The BIOS does NOT report the MCFG addresses 
as reserved in the E820 address map. However, the addresses ARE claimed as 
reserved via the ACPI motherboard resource devnode (PNP0C01) mechanism.

On this system, Xen ignores the MCFG table as it fails the E820 check. dom0 
during early boot does the same thing. However, once ACPI driver is online, 
dom0 validates the MCFG table against the motherboard resource devnode and then 
accepts the MCFG table. You now end up with a situation where dom0 can 
correctly enumerate and use PCIe devices but Xen cannot access extended 
configuration space.

If an attempt is now made to pass-through a PCIe SRIOV VF with MSI interrupt 
for e.g., you see following warning (and IIRC, the passed through device did 
not work):
(XEN) Xen WARN at msi.c:688
(XEN) ----[ Xen-4.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    7
(XEN) RIP:    e008:[<ffff82c48015a9c5>] pci_enable_msi+0x645/0x8f0
(XEN) RFLAGS: 0000000000010206   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: 00000000fa160000   rcx: 0000000000000000
(XEN) rdx: 0000000000000cfe   rsi: 0000000000000246   rdi: ffff82c48023db80
(XEN) rbp: ffff830413e97ea8   rsp: ffff830413e97d28   r8:  0000000000000100
(XEN) r9:  ffff830413e97c7c   r10: 0000000000000000   r11: 0000000000000070
(XEN) r12: 0000000000000481   r13: ffff8304079b6250   r14: ffff830413e97e28
(XEN) r15: 0000000000000003   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 0000000413e69000   cr2: 00000000d84341b8
(XEN) ds: 007b   es: 007b   fs: 00d8   gs: 0033   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff830413e97d28:
(XEN)    ffff830407bcdf50 ffff8300de982000 ffff8304079b62e8 0000000000000040
(XEN)    ffff82c400000003 0000000000000064 ffff83041f480fd8 00000070c0027631
(XEN)    ffff8303b1471e80 0000001000000004 0000007200000001 00000000000fa160
(XEN)    00000000000fa160 0000000000000000 00000000fa160000 0000000000000033
(XEN)    0000000100000000 0000000300000000 ffff8304108441d8 ffff830413e97ea8
(XEN)    0000000000000033 ffff830407a84000 000000000000002a 00000000ffffffed
(XEN)    ffff8304079b6250 ffff82c48015c63c 0000000000000000 00000000000000cc
(XEN)    000000000000002a 00000000000000a8 ffff83041f402a80 0000000000000000
(XEN)    00000000ffffffff ffff8304123840b8 00000000088c6004 0000000000000033
(XEN)    000000000000002a ffff830413e97e78 ffff830413e97ea8 ffff82c4801fb561
(XEN)    0000000400000000 00000000c02f6361 0000000000000001 ffffffffffffffff
(XEN)    0000008100000004 fa16000000000000 ffff830400000000 0000000000000000
(XEN)    0000008100000004 000000000000002a 00000000fa160000 ffff830407a84000
(XEN)    00000000000002ed 00c0bb0013e97ee0 0000000000000000 ffff8300deff2000
(XEN)    00000000d6bbfed4 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 ffff82c4801fd324 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 00000000d6bbfed4 000000000000000d
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000021 00000000088c6004 00000000bf9006f8 00000000b773f658
(XEN)    00000000b7730c28 0000010000000000 00000000c0101427 0000000000000061
(XEN) Xen call trace:
(XEN)    [<ffff82c48015a9c5>] pci_enable_msi+0x645/0x8f0
(XEN)    [<ffff82c48015c63c>] map_domain_pirq+0x25c/0x350
(XEN)    [<ffff82c4801fb561>] compat_physdev_op+0xe31/0x12f0
(XEN)    [<ffff82c4801fd324>] compat_hypercall+0x74/0x80


Is this machine an exception? As we move forward, are we likely to see more 
such systems (relying more on ACPI and less on E820 for reserving the MCFG 
range)? Is it worth adding a mmcfg=force option to xen to ignore E820 
validation result?

Regards,
Santosh


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel