RE: [Xen-devel] IRQ SMP affinity problems in domU with vcpus > 4 on HP ProLiant G6 with dual Xeon 5540 (Nehalem)

["Cinco, Dante" <Dante.Cinco@xxxxxxx>]

Qing,

I'm attaching a tar'd directory that contains the various log files I gathered 
from my system. When I tried adding "hvm_debug=0x200" in the Xen command line, 
the domU became unaccessible on boot up with the Xen console constantly 
printing this message: "(XEN) [HVM:1.0] <vioapic_irq_positive_edge> irq 2." So 
I backed out the hvm_debug but hopefully there are still enough logging to 
provide some clues. Here's a summary of the events leading to the lost 
interrupts:

Boot Xen 3.5-unstable with 2.6.30.3
- command line: /xen-3.5-unstable.gz com1=115200,8n1 console=com1 acpi=force 
apic=on iommu=1,no-intremap,passthrough loglvl=all loglvl_guest=all
- command line: module /vmlinuz-2.6.31.1 root=UUID=xxx ro 
pciback.hide=(07:00.0)(07:00.1)(07:00.2)(07:00.3) acpi=force console=ttyS0
- dom0: lspci -vv shows device at IRQ 32 with MSI message address, data = 0x0, 
0x0

Bringup domU with vcpus=5, hap=0, 
pci=['07:00.0@8','07:00.1@9','07:00.2@a','07:00.3@b'] (device driver not yet 
loaded)
- dom0: lspci -vv shows device at IRQ 32 (07:00.0) with MSI message address, 
data = 0xfee01000, 0x407b
- config: 

Load kernel module that contains device driver
- dom0: no change in lspci -vv
- domU: lspci -vv shows device at IRQ 48 (00:08.0) with MSI message address, 
data = 0xfee00000, 0x4059
- domU: /proc/interrupts show interrupts for IRQ 48 going to CPU0

Change /proc/irq/48/smp_affinity from 1f to 1
- dom0: no change to lspci -vv
- domU: no change to lspci -vv
- domU: /proc/interrupts show interrupts for IRQ 48 going to CPU0

Change /proc/irq/48/smp_affinity from 1 to 2
- dom0: lspci -vv shows MSI message data changed from 0x407b to 0x40d3, address 
the same
- domU: lspci -vv shows new MSI message address, data = 0xfee02000, 0x4079
- domU: no more interrupts from IRQ 48
- Xen console: (XEN) do_IRQ: 8.211 No irq handler for vector (irq -1)

Dante

-----Original Message-----
From: Qing He [mailto:qing.he@xxxxxxxxx] 
Sent: Friday, October 09, 2009 2:08 AM
To: Keir Fraser
Cc: Cinco, Dante; xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-devel] IRQ SMP affinity problems in domU with vcpus > 4 on HP 
ProLiant G6 with dual Xeon 5540 (Nehalem)

On Fri, 2009-10-09 at 05:35 +0800, Keir Fraser wrote:
> On 08/10/2009 19:11, "Cinco, Dante" <Dante.Cinco@xxxxxxx> wrote:
> 
> > The IRQ SMP affinity problem happens on just the passthrough one using MSI.
> > 
> > I've only used Xen 3.4.1. Are you aware of recent code changes that 
> > may address this issue?
> 
> No, but it might be worth a try. Unfortunately I'm not so familiar 
> with the MSI passthru code as I am with the rest of the irq emulation 
> layer. Qing He
> (cc'ed) may be able to assist, as I think he did much of the 
> development of MSI support for passthru devices.
> 

MSI passthru uses emulation, there is nothing to do between the guest affinity 
and the physical affinity. When an MSI is received, a vmsi logic calculates the 
destination and sets the virtual local APIC of that VCPU.

But after checking the code, the part handling DM=0 is there and I haven't 
found big problems on the first glance, maybe there is some glitch that causes 
the MSI failure in physical mode.

Some debug log can be helpful to track down the problem. Can you add 
'hvm_debug=0x200' to the xen command line and post the xm dmesg result?
This will print hvm debug level DBG_LEVEL_IOAPIC which includes vmsi delivery 
logic.

There are two patches between 3.4.1 and unstable (20084, 20140), these are 
mainly cleanup patches but the related code does change, don't know if they fix 
this issue.

Thanks,
Qing

Attachment: irq_smp_affinity_problem.tar.gz
Description: irq_smp_affinity_problem.tar.gz

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel