|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem
On Mon, Sep 27, 2010 at 12:54 PM, Konrad Rzeszutek Wilk
<konrad.wilk@xxxxxxxxxx> wrote:
> On Mon, Sep 27, 2010 at 12:16:50PM -0700, Bruce Edge wrote:
>> On Mon, Sep 27, 2010 at 10:24 AM, Konrad Rzeszutek Wilk
>> <konrad.wilk@xxxxxxxxxx> wrote:
>> >
>> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
>> > > One of our developers who is working on a tachyon driver is
>> > > complaining that the pvops domU kernel is not working for these MSI
>> > > interrupts.
>> > > This is using the current head of xen/2.6.32.x on both a single
>> > > Nahelam 920 and a dual E5540. This behavior is consistent with Xen
>> > > 4.0.1, 4.0.2.rc1-pre and 4.1.
>> > >
>> > > Here are his comments:
>> > >
>> > > - the driver has no problem to enable msi interrupt and request the
>> > > interrupt through kernel functions pci_enable_msi & request_irq
>> >
>> > What shows up in the Xen console when you send the 'q' key? Does it
>> > show that the vector is assigned to the appropiate guest?
>>
>> The Xen console q key shows that the domU is assigned:
>>
>> (XEN) Interrupts { 32, 41-42, 47 }
>
> Aha!
>
>>
>> but the domU thinks it has:
>>
>> 124/125/126/127
>>
>> Is there some mapping that's taking place, or is this plain wrong?
>
> That looks wrong. The IRQ numbers (even though they are MSI vectors) are
> setup as IRQ numbers in the DomU guest. You should have seen
>
> 32:
> 41:
> 42:
> 47:
> in you /proc/interrupts on your DomU guest.
>
> I wonder what broke - can you use
> git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
> devel/xen-pcifront-0.5 (or pv/pcifront-2.6.32)?
Please forgive the git ignorance.
Is this the right syntax?
git clone
git://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.32
linux-2.6.32-pv-pcifront
Initialized empty Git repository in
/import/kaan/bedge/src/xen/kernel/pv-ops/linux-2.6.32-pv-pcifront/.git/
fatal: The remote end hung up unexpectedly
Or:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
Initialized empty Git repository in
/import/kaan/bedge/src/xen/kernel/pv-ops/xen/.git/
remote: error: Could not read 59eab2f8f04147c5aadc99f2034ca7e5b81e890f
remote: fatal: Failed to traverse parents of commit
979e121cb348add17ed8171bf447b27a3a9d1be3
remote: aborting due to possible repository corruption on the remote side.
fatal: early EOF
fatal: index-pack failed
>
> It has the latest pcifront driver but without the PVonHVM enhancments
> so we can try to eliminate the PvONHVM logic out of the picture.
>
>>
>> >
>> > > - the interrupt does happen. But the interrupt service routine of
>> > > tachyon driver doesn't detect any interrupt status related to this
>> > > interrupt, which inhibits the tachyon chip from coming on-line. And
>> > > there are high count of tachyon interrupt in /proc/interrupts
>> >
>> > Is it checking the PCI_STATUS_INTERRUPT or the appropiate register
>> > in the MMIO BAR?
>> >
>>
>> The driver would check the appropriate register (tachyon registers) in
>> the MMIO to determine the source of interrupts.
>
> OK, so that isn't it. Is there anything at these vectors:
> 7c, 7d, 7e, and 7f? When you use xen debug-keys 'i' or 'q' it should give you
> an inkling what device this is set for.
When I run a distro kernel in hvm mode, I get the expected irq mappings:
'i' - Note 66 - 69
(XEN) IRQ: 66 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:3a
type=PCI-MSI status=00000010 in-flight=0
domain-list=10:127(----),
(XEN) IRQ: 67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:42
type=PCI-MSI status=00000010 in-flight=0
domain-list=10:126(----),
(XEN) IRQ: 68 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:4a
type=PCI-MSI status=00000010 in-flight=0
domain-list=10:125(----),
(XEN) IRQ: 69 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:52
type=PCI-MSI status=00000010 in-flight=0
domain-list=10:124(----)
'q'
(XEN) Interrupts { 32, 41-42, 47, 124-127 }
The same data with pv-ops kernel shows:
'i'
IRQ numbers stop at 65, no 66 - 69 present:
(XEN) IRQ: 63 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:91
type=PCI-MSI status=00000010 in-flight=0
domain-list=0:289(----),
(XEN) IRQ: 64 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:99
type=PCI-MSI status=00000002 mapped, unbound
(XEN) IRQ: 65 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b1
type=PCI-MSI status=00000010 in-flight=0
domain-list=0:287(----),
(XEN) IO-APIC interrupt information:
'q'
(XEN) Interrupts { 32, 41-42, 47 }
>
>>
>> > >
>> > > kaan-18-dpm:~# cat /proc/interrupts | grep TACH
>> > > 124: 760415 0 0 0 0 0
>> > > 0 0 0 0 0 0
>> > > 0 0 xen-pirq-pcifront-msi HW_TACHYON
>> > > 125: 762234 0 0 0 0 0
>> > > 0 0 0 0 0 0
>> > > 0 0 xen-pirq-pcifront-msi HW_TACHYON
>> > > 126: 764180 0 0 0 0 0
>> > > 0 0 0 0 0 0
>> > > 0 0 xen-pirq-pcifront-msi HW_TACHYON
>> > > 127: 764164 0 0 0 0 0
>> > > 0 0 0 0 0 0
>> > > 0 0 xen-pirq-pcifront-msi HW_TACHYON
>> >
>> > Can you provide the full dmesg output?
>>
>> Attached.
>>
>> Some possibly related messages on dom0 console:
>>
>> [ 1882.269778] pciback 0000:07:00.0: enabling device (0000 -> 0003)
>> [ 1882.269800] xen: registering gsi 32 triggering 0 polarity 1
>> [ 1882.269827] xen_allocate_pirq: returning irq 32 for gsi 32
>> [ 1882.269834] xen: --> irq=32
>> [ 1882.269841] Already setup the GSI :32
>> [ 1882.269847] pciback 0000:07:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ
>> 32
>> [ 1882.269866] pciback 0000:07:00.0: setting latency timer to 64
>> [ 1882.270463] pciback 0000:07:00.0: Driver tried to write to a
>> read-only configuration space field at offset 0x62, size 2. This may
>> be harmless, but if you have problems with your device:
>
> Uhhh, for that I think you need to do 'lspci -vvv -xxx -s 07:00.00'
> to find out what is at the configuration space. You could enable
> it using the permissive attribute.
>
>> [ 1882.270465] 1) see permissive attribute in sysfs
>> [ 1882.270467] 2) report problems to the xen-devel mailing list along
>> with details of your device obtained from lspci.
>> [ 1882.270615] alloc irq_desc for 478 on node 0
>> [ 1882.270625] alloc kstat_irqs on node 0
>
> So for 478: what do you see? xen-pciback I presume?
>> [ 1882.348411] pciback 0000:07:00.1: enabling device (0000 -> 0003)
>> [ 1882.348433] xen: registering gsi 42 triggering 0 polarity 1
>> [ 1882.348440] xen_allocate_pirq: returning irq 42 for gsi 42
>> [ 1882.348445] xen: --> irq=42
>> [ 1882.348472] Already setup the GSI :42
>> [ 1882.348479] pciback 0000:07:00.1: PCI INT B -> GSI 42 (level, low) -> IRQ
>> 42
>> [ 1882.348497] pciback 0000:07:00.1: setting latency timer to 64
>> [ 1882.349063] pciback 0000:07:00.1: Driver tried to write to a
>> read-only configuration space field at offset 0x62, size 2. This may
>> be harmless, but if you have problems with your device:
>> [ 1882.349066] 1) see permissive attribute in sysfs
>> [ 1882.349067] 2) report problems to the xen-devel mailing list along
>> with details of your device obtained from lspci.
>> [ 1882.349205] alloc irq_desc for 477 on node 0
>> [ 1882.349215] alloc kstat_irqs on node 0
>> [ 1882.402893] pciback 0000:07:00.2: enabling device (0000 -> 0003)
>> [ 1882.402908] xen: registering gsi 47 triggering 0 polarity 1
>> [ 1882.402913] xen_allocate_pirq: returning irq 47 for gsi 47
>> [ 1882.402916] xen: --> irq=47
>> [ 1882.402921] Already setup the GSI :47
>> [ 1882.402925] pciback 0000:07:00.2: PCI INT C -> GSI 47 (level, low) -> IRQ
>> 47
>> [ 1882.402938] pciback 0000:07:00.2: setting latency timer to 64
>> [ 1882.403280] pciback 0000:07:00.2: Driver tried to write to a
>> read-only configuration space field at offset 0x62, size 2. This may
>> be harmless, but if you have problems with your device:
>> [ 1882.403282] 1) see permissive attribute in sysfs
>> [ 1882.403282] 2) report problems to the xen-devel mailing list along
>> with details of your device obtained from lspci.
>> [ 1882.403380] alloc irq_desc for 476 on node 0
>> [ 1882.403386] alloc kstat_irqs on node 0
>> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
>> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
>> fault addr e6f80000, iommu reg = ffff82c3fff57000
>> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
>> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = e6f80
>> (XEN) root_entry = ffff83019ff70000
>> (XEN) root_entry[7] = 19cf52001
>> (XEN) context = ffff83019cf52000
>> (XEN) context[0] = 102_706dc005
>> (XEN) l4 = ffff8300706dc000
>> (XEN) l4_index = 0
>> (XEN) l4[0] = 706db003
>> (XEN) l3 = ffff8300706db000
>> (XEN) l3_index = 3
>> (XEN) l3[3] = 702b6003
>> (XEN) l2 = ffff8300702b6000
>> (XEN) l2_index = 137
>> (XEN) l2[137] = 0
>> (XEN) l2[137] not present
>> (XEN) traps.c:466:d0 Unhandled nmi fault/trap [#2] on VCPU 0 [ec=0000]
>
> That is not good. What changed from your earlier emails that this was
> triggered?
Nothing
> Or was it triggered all along?
Yes, I just included it for completeness
> What happens if you run the system without the iommu enabled?
Haven't tried yet. Will check that next.
-Bruce
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |