[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] PVH Dom0 Intel IOMMU issues



On Mon, Apr 17, 2017 at 09:03:12AM +0800, Chao Gao wrote:
> On Mon, Apr 17, 2017 at 08:47:48AM +0100, Roger Pau Monné wrote:
> >On Mon, Apr 17, 2017 at 07:32:45AM +0800, Chao Gao wrote:
> >> On Fri, Apr 14, 2017 at 04:34:41PM +0100, Roger Pau Monné wrote:
> >> >Hello,
> >> >
> >> >Although PVHv2 Dom0 is not yet finished, I've been trying the current 
> >> >code on
> >> >different hardware, and found that with pre-Haswell Intel hardware PVHv2 
> >> >Dom0
> >> >completely freezes the box when calling iommu_hwdom_init in 
> >> >dom0_construct_pvh.
> >> >OTOH the same doesn't happen when using a newer CPU (ie: haswell or 
> >> >newer).
> >> >
> >> >I'm not able to debug that in any meaningful way because the box seems to 
> >> >lock
> >> >up completely, even the watchdog NMI stops working. Here is the boot log, 
> >> >up to
> >> >the point where it freezes:
> >> 
> >> I try "dom0=pvh" with my skylake. An assertion failed. Is it a software 
> >> bug?
> >> 
> >---8<---
> >diff --git a/xen/arch/x86/hvm/vioapic.c b/xen/arch/x86/hvm/vioapic.c
> >index 527ac2aadd..1df7710041 100644
> >--- a/xen/arch/x86/hvm/vioapic.c
> >+++ b/xen/arch/x86/hvm/vioapic.c
> >@@ -625,6 +625,9 @@ int vioapic_init(struct domain *d)
> >         nr_gsis += nr_pins;
> >     }
> > 
> >+    printk("domain nr_gsis: %u vioapic gsis: %u nr_irqs_gsi: %u 
> >highest_gsi: %u\n",
> >+           hvm_domain_irq(d)->nr_gsis, nr_gsis, nr_irqs_gsi, highest_gsi());
> >+
> >     ASSERT(hvm_domain_irq(d)->nr_gsis == nr_gsis);
> > 
> >     d->arch.hvm_domain.nr_vioapics = nr_vioapics;
> 
> With the above patch,
> (XEN) [   10.420001] PCI: MCFG area at 80000000 reserved in E820
> (XEN) [   10.426854] PCI: Using MCFG for segment 0000 bus 00-ff
> (XEN) [   10.433952] Intel VT-d iommu 6 supported page sizes: 4kB, 2MB, 1GB.
> (XEN) [   10.441856] Intel VT-d iommu 5 supported page sizes: 4kB, 2MB, 1GB.
> (XEN) [   10.449759] Intel VT-d iommu 4 supported page sizes: 4kB, 2MB, 1GB.
> (XEN) [   10.457671] Intel VT-d iommu 3 supported page sizes: 4kB, 2MB, 1GB.
> (XEN) [   10.465585] Intel VT-d iommu 2 supported page sizes: 4kB, 2MB, 1GB.
> (XEN) [   10.473485] Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
> (XEN) [   10.481394] Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
> (XEN) [   10.489299] Intel VT-d iommu 7 supported page sizes: 4kB, 2MB, 1GB.
> (XEN) [   10.497196] Intel VT-d Snoop Control enabled.
> (XEN) [   10.503196] Intel VT-d Dom0 DMA Passthrough not enabled.
> (XEN) [   10.510145] Intel VT-d Queued Invalidation enabled.
> (XEN) [   10.516646] Intel VT-d Interrupt Remapping enabled.
> (XEN) [   10.523173] Intel VT-d Posted Interrupt not enabled.
> (XEN) [   10.529775] Intel VT-d Shared EPT tables enabled.
> (XEN) [   10.548815] I/O virtualisation enabled
> (XEN) [   10.554186]  - Dom0 mode: Relaxed
> (XEN) [   10.559264] Interrupt remapping enabled
> (XEN) [   10.564854] nr_sockets: 5
> (XEN) [   10.569231] Enabled directed EOI with ioapic_ack_old on!
> (XEN) [   10.577294] ENABLING IO-APIC IRQs
> (XEN) [   10.582245]  -> Using old ACK method
> (XEN) [   10.587967] ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
> (XEN) [   10.797645] TSC deadline timer enabled
> (XEN) [   10.887286] Defaulting to alternative key handling; send 'A' to 
> switch to normal mode.
> (XEN) [   10.897864] mwait-idle: MWAIT substates: 0x2020
> (XEN) [   10.899335] mwait-idle: v0.4.1 model 0x55
> (XEN) [   10.900799] mwait-idle: lapic_timer_reliable_states 0xffffffff
> (XEN) [   10.902304] VMX: Supported advanced features:
> (XEN) [   10.903781]  - APIC MMIO access virtualisation
> (XEN) [   10.905258]  - APIC TPR shadow
> (XEN) [   10.907138]  - Extended Page Tables (EPT)
> (XEN) [   10.908782]  - Virtual-Processor Identifiers (VPID)
> (XEN) [   10.910262]  - Virtual NMI
> (XEN) [   10.911719]  - MSR direct-access bitmap
> (XEN) [   10.913188]  - Unrestricted Guest
> (XEN) [   10.914650]  - APIC Register Virtualization
> (XEN) [   10.916126]  - Virtual Interrupt Delivery
> (XEN) [   10.917596]  - Posted Interrupt Processing
> (XEN) [   10.919066]  - VMCS shadowing
> (XEN) [   10.920519]  - VM Functions
> (XEN) [   10.921976]  - Virtualisation Exceptions
> (XEN) [   10.923448]  - Page Modification Logging
> (XEN) [   10.924918]  - TSC Scaling
> (XEN) [   10.926371] HVM: ASIDs enabled.
> (XEN) [   10.927829] HVM: VMX enabled
> (XEN) [   10.929278] HVM: Hardware Assisted Paging (HAP) detected
> (XEN) [   10.930762] HVM: HAP page sizes: 4kB, 2MB, 1GB
> (XEN) [    0.000000] CMCI: threshold 0x2 too large for CPU56 bank 6, using 0x1
> (XEN) [    0.000000] CMCI: threshold 0x2 too large for CPU56 bank 9, using 0x1
> (XEN) [    0.000000] CMCI: threshold 0x2 too large for CPU56 bank 10, using 
> 0x1
> (XEN) [    0.000000] CMCI: threshold 0x2 too large for CPU56 bank 11, using 
> 0x1
> (XEN) [   13.216648] Brought up 112 CPUs
> (XEN) [   13.739330] build-id: dc4540250abe5d96614d340c67069e390c37c21c
> (XEN) [   13.740816] Running stub recovery selftests...
> (XEN) [   13.742258] traps.c:3466: GPF (0000): ffff82d0bffff041 
> [ffff82d0bffff041] -> ffff82d080359cf2
> (XEN) [   13.745155] traps.c:813: Trap 12: ffff82d0bffff040 
> [ffff82d0bffff040] -> ffff82d080359cf2
> (XEN) [   13.748046] traps.c:1215: Trap 3: ffff82d0bffff041 
> [ffff82d0bffff041] -> ffff82d080359cf2
> (XEN) [   13.771452] TSC warp detected, disabling TSC_RELIABLE
> (XEN) [   13.772897] ACPI sleep modes: S3
> (XEN) [   13.774329] VPMU: disabled
> (XEN) [   13.775757] mcheck_poll: Machine check polling timer started.
> (XEN) [   13.777246] Dom0 has maximum 1448 PIRQs
> (XEN) [   13.779426] domain nr_gsis: 104 vioapic gsis: 88 nr_irqs_gsi: 104 
> highest_gsi: 103

It seems like we are not properly adding/accounting the vIO APICs, but I cannot
really see how. I have another patch for you to try below.

Thanks, Roger.

---8<---
        diff --git a/xen/arch/x86/hvm/vioapic.c b/xen/arch/x86/hvm/vioapic.c
index 527ac2aadd..40075e2756 100644
--- a/xen/arch/x86/hvm/vioapic.c
+++ b/xen/arch/x86/hvm/vioapic.c
@@ -610,11 +610,15 @@ int vioapic_init(struct domain *d)
            xzalloc_array(struct hvm_vioapic *, nr_vioapics)) == NULL) )
         return -ENOMEM;
 
+    printk("Adding %u vIO APICs\n", nr_vioapics);
+
     for ( i = 0; i < nr_vioapics; i++ )
     {
         unsigned int nr_pins = is_hardware_domain(d) ? nr_ioapic_entries[i] :
             ARRAY_SIZE(domain_vioapic(d, 0)->domU.redirtbl);
 
+        printk("vIO APIC %u has %u pins\n", i, nr_pins);
+
         if ( (domain_vioapic(d, i) =
               xmalloc_bytes(hvm_vioapic_size(nr_pins))) == NULL )
         {
@@ -623,8 +627,12 @@ int vioapic_init(struct domain *d)
         }
         domain_vioapic(d, i)->nr_pins = nr_pins;
         nr_gsis += nr_pins;
+        printk("nr_gsis: %u\n", nr_gsis);
     }
 
+    printk("domain nr_gsis: %u vioapic gsis: %u nr_irqs_gsi: %u highest_gsi: 
%u\n",
+           hvm_domain_irq(d)->nr_gsis, nr_gsis, nr_irqs_gsi, highest_gsi());
+
     ASSERT(hvm_domain_irq(d)->nr_gsis == nr_gsis);
 
     d->arch.hvm_domain.nr_vioapics = nr_vioapics;


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.