Re: [Xen-devel] dom0 kernel - irq nobody cared ... the continuing saga ..

[Sander Eikelenboom <linux@xxxxxxxxxxxxxx>]

Tuesday, February 10, 2015, 2:26:43 PM, you wrote:

>>>> On 10.02.15 at 14:07, <linux@xxxxxxxxxxxxxx> wrote:
>> Tuesday, February 10, 2015, 10:35:36 AM, you wrote:
>>>>>> On 10.02.15 at 10:19, <linux@xxxxxxxxxxxxxx> wrote:
>>>>> I would have thought that xen-pciback would install an interrupt
>>>>> handler here too when a device using IRQ 18 gets handed to a
>>>>> guest. May there be something broken in xen_pcibk_control_isr()?
>>>> 
>>>> It seems to only do that for PV guests, not for HVM.
>> 
>>> Interesting; no idea why that would be.
>> 
>> Coming back to this part .. with some additional debugging on HVM guest 
>> start i 
>> get:
>> [   84.362097] pciback 0000:02:00.0: resetting (FLR, D3,  bus) the device
>> [   84.422884] pciback 0000:02:00.0: xen_pcibk_reset_device: me here
>> [   84.452639] pciback 0000:02:00.0: xen_pcibk_control_isr: return !enable 
>> && 
>> !dev_data->isr_on enable:0 dev_data->isr_on:0
>> [   85.636725] pciback 0000:00:19.0: resetting (FLR, D3,  ) the device
>> [   85.774845] pciback 0000:00:19.0: xen_pcibk_reset_device: me here
>> [   85.810857] pciback 0000:00:19.0: xen_pcibk_control_isr: return !enable 
>> && 
>> !dev_data->isr_on enable:0 dev_data->isr_on:0
>> [   85.865075] xen_pciback: vpci: 0000:02:00.0: assign to virtual slot 0
>> [   85.886926] pciback 0000:02:00.0: registering for 1
>> [   85.907936] xen_pciback: vpci: 0000:00:19.0: assign to virtual slot 1
>> [   85.929490] pciback 0000:00:19.0: registering for 1
>>  
>> So it bails out on:
>>         /* Asked to disable, but ISR isn't runnig */
>>         if (!enable && !dev_data->isr_on){
>>                 dev_warn(&dev->dev, "%s: return !enable && !dev_data->isr_on 
>> enable:%d dev_data->isr_on:%d\n", __func__, enable, dev_data->isr_on ? 1 : 
>> 0);
>>                 return;
>>         }

> I don't really know how this code is supposed to work (we don't use
> it in our kernels), but it seems the more interesting question is what
> happens when xen_pcibk_do_op() calls this function. I also note there
> is a sysfs interface to control some aspects of this, but I can't see how
> that would work when it only sets ->isr_on but doesn't install the IRQ
> handler if it isn't already installed.

> Jan

Suse is still forward porting and not considering upstream/pvops ?

Hmmm it seems xen_pcibk_do_op() is never called ..

Turned up the pciback debug, and annotated xen_pcibk_do_op() it self with 
more warnings, it should print something when invoked.
complete dmesg attached.

Here the debug output from pciback:

seizing:
[    9.113134] xen_pciback: wants to seize 0000:02:00.0
[    9.113137] xen_pciback: wants to seize 0000:00:19.0
[    9.113146] pciback 0000:00:00.0: probing...
[    9.113154] pciback 0000:00:02.0: probing...
[    9.113160] pciback 0000:00:14.0: probing...
[    9.113165] pciback 0000:00:16.0: probing...
[    9.113171] pciback 0000:00:16.3: probing...
[    9.113176] pciback 0000:00:19.0: probing...
[    9.113178] pciback 0000:00:19.0: seizing device
[    9.113182] pciback 0000:00:19.0: pcistub_device_alloc
[    9.113184] pciback 0000:00:19.0: deferring initialization
[    9.113189] pciback 0000:00:1a.0: probing...
[    9.113194] pciback 0000:00:1b.0: probing...
[    9.113200] pciback 0000:00:1c.0: probing...
[    9.113205] pciback 0000:00:1c.2: probing...
[    9.113210] pciback 0000:00:1d.0: probing...
[    9.113216] pciback 0000:00:1f.0: probing...
[    9.113221] pciback 0000:00:1f.2: probing...
[    9.113226] pciback 0000:00:1f.3: probing...
[    9.113232] pciback 0000:02:00.0: probing...
[    9.113234] pciback 0000:02:00.0: seizing device
[    9.113237] pciback 0000:02:00.0: pcistub_device_alloc
[    9.113239] pciback 0000:02:00.0: deferring initialization

initialization while dom0 is still booting:
[   14.371824] pciback 0000:02:00.0: initializing...
[   14.371829] pciback 0000:02:00.0: initializing config
[   14.371879] pciback 0000:02:00.0: enabling device
[   14.371944] xen: registering gsi 18 triggering 0 polarity 1
[   14.371949] Already setup the GSI :18
[   14.372457] pciback 0000:02:00.0: save state of device
[   14.372582] pciback 0000:02:00.0: resetting (FLR, D3, etc) the device
[   14.403788] pciback 0000:02:00.0: reset device
[   14.403794] pciback 0000:02:00.0: xen_pcibk_reset_device: me heres
[   14.404638] pciback 0000:02:00.0: xen_pcibk_control_isr: return !enable && 
!dev_data->isr_on enable:0 dev_data->isr_on:0
[   14.406186] pciback 0000:00:19.0: initializing...
[   14.406189] pciback 0000:00:19.0: initializing config
[   14.406225] pciback 0000:00:19.0: enabling device
[   14.406320] xen: registering gsi 20 triggering 0 polarity 1
[   14.406340] xen: --> pirq=20 -> irq=20 (gsi=20)
[   14.406372] pciback 0000:00:19.0: save state of device
[   14.406435] pciback 0000:00:19.0: resetting (FLR, D3, etc) the device
[   14.507741] pciback 0000:00:19.0: reset device
[   14.507747] pciback 0000:00:19.0: xen_pcibk_reset_device: me heres
[   14.538842] pciback 0000:00:19.0: xen_pcibk_control_isr: return !enable && 
!dev_data->isr_on enable:0 dev_data->isr_on:0
[   14.571345] xen_pciback: backend is vpci

guest start:
[  117.061148] pciback 0000:02:00.0: resetting (FLR, D3,  bus) the device
[  117.092324] pciback 0000:02:00.0: OK
[  117.123935] pciback 0000:02:00.0: xen_pcibk_reset_device: me heres
[  117.153042] pciback 0000:02:00.0: xen_pcibk_control_isr: return !enable && 
!dev_data->isr_on enable:0 dev_data->isr_on:0
[  118.336807] pciback 0000:00:19.0: resetting (FLR, D3,  ) the device
[  118.371580] pci 0000:00:00.0: not owned by us!
[  118.475884] pciback 0000:00:19.0: xen_pcibk_reset_device: me heres
[  118.497516] pciback 0000:00:19.0: xen_pcibk_control_isr: return !enable && 
!dev_data->isr_on enable:0 dev_data->isr_on:0
[  118.581239] xen-pciback pci-1-0: allocated pdev @ 0xffff88000007e9c0
[  118.582466] xen-pciback pci-1-0: getting be setup
[  118.582683] xen-pciback pci-1-0: exporting dom 0 bus 2 slot 0 func 0
[  118.582688] xen_pciback: vpci: 0000:02:00.0: assign to virtual slot 0
[  118.613355] pciback 0000:02:00.0: registering for 1
[  118.635176] xen-pciback pci-1-0: exporting dom 0 bus 0 slot 19 func 0
[  118.635180] xen_pciback: vpci: 0000:00:19.0: assign to virtual slot 1
[  118.657663] pciback 0000:00:19.0: registering for 1
[  118.682808] xen-pciback pci-1-0: Publishing pci roots
[  118.682907] xen-pciback pci-1-0: writing root 0 at 0000:00
[  118.686896] xen-pciback pci-1-0: fe state changed 1



So this one is up to Konrad i guess ...

I haven't checked the call chain of xen_pcibk_do_op .. but that could be a
side effect of libxl not imitating pci-front good enough (since HVM guests
don't use the pci-front driver, but instead rely on libxl and Qemu to play
those parts.

One of the issues i already mentioned that devices never get to the state 
connected, for HVM guests. The backend state stays 3 (XenbusStateInitialised)
and the frontend state stays 1 (XenbusStateInitialising).

Until now i thought is was benign and only causing issues on shutdown:
   - not cleaning up of the device in the guest itself.  which causes invoking
     the fallbacks of which a few were fixed just before the 4.5.0 release.
   - issues when only removing one assigned device of multiple assigned to a 
     guest.
since the passed through devices in principle are working OK.
 
--
Sander

Attachment: dmesg.txt
Description: Text document

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel