[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] xen panics when setting int3 traps
Add Guangrong who will be looking at nested virtualization in our side. > From: Tim Deegan [mailto:tim@xxxxxxx] > Sent: Friday, December 11, 2015 6:21 PM > > Hi, > > Thanks for the report. The maintainers list changed recently, so I'm > CC'ing the current set for VMX and x86/mm. > > Since this is an L1 xen crash, the bug is probably in the code that > tried to inject the trap (vmi_write_8_pa() and callers). It's > interesting that both the L1 crashes were in unlock routines, but that > could be a coincidence. > > At 11:13 +0800 on 11 Dec (1449832424), quizy_jones@xxxxxxxxxxx wrote: > > The reproduce method is a little complex. We rely on libvmi to translate > > virtual addess > into physical address and inject traps. > > > > Goal: monitor all hypercalls of L1 xen (4.4.1) from dom0 of L0 xen (4.4.6) > > 1. obtain virtual address of hypercall handler (GVA) from xen-syms-4.4.1 > > file > > 2. use vmi_pagetable_lookup of libvmi api to translate GVA into guest > > physical address > (GPA) > > 3. inject int3 into the first byte of GPA using libvmi api vmi_write_8_pa > > 4. listen on vmexit events caused by #BP > > The code of the above procedures is here. > > By running 'hvm10 nested-xen' and create/destroy any domains in nested-xen, > > you can > reproduce the error. (nested-xen is the domain name of L1 xen) > > > > My doubt is whether the second step is right. I.E. How xen manages > > its own memory translations in nested virtualization? And does the > > L0 xen have privilege to write the memory of L1 xen? BTW, I'm using > > (v)EPT for nested xen. > > Yes, L0 Xen can write to L1 Xen's memory. Internally, Xen has an EPT > table for L1 Xen, and mainatins EPT tables that mirror the L1's EPT > tables (but with the extra translation) for the CPU to use when > running L2 guests. The code is in xen/arch/x86/mm/hap/nested_*. > > The next step is to log everything that goes on around that > vmi_write_8_pa() call, all the addresses it uses and what data it > changes, and also modify the L1 hypervisor to print the contents of > the target in a debug keyhandler, so you can check whether > vmi_write_8_pa() did what you expected. > > Cheers, > > Tim. > > > > > Jones > > > > From: Konrad Rzeszutek Wilk > > Date: 2015-12-11 10:17 > > To: quizy_jones@xxxxxxxxxxx > > CC: xen-devel > > Subject: Re: [Xen-devel] xen panics when setting int3 traps > > On Fri, Dec 11, 2015 at 10:01:13AM +0800, quizy_jones@xxxxxxxxxxx wrote: > > > Aslo reboot when destroy a VM. > > > > You need to give more context on how to reproduce this, and you should > > also CC the maintainers of the code. Please look in MAINTAINERS file. > > > > > > > > The logs from L1 Xen: > > > (XEN) ----[ Xen-4.4.1 x86_64 debug=n Not tainted ]---- > > > (XEN) CPU: 0 > > > (XEN) RIP: e008:[<ffff82d080127b6b>] _spin_unlock+0x1b/0x30 > > > (XEN) RFLAGS: 0000000000010202 CONTEXT: hypervisor > > > (XEN) rax: ffff82d0802f8320 rbx: 00007f4b024a9004 rcx: > > > 0000000000000002 > > > (XEN) rdx: ffff82d0802b0000 rsi: 0000000000000080 rdi: > > > 000000007c6ebdb0 > > > (XEN) rbp: ffff82d0802b7e48 rsp: ffff82d0802b7dc0 r8: > > > 0000000000000004 > > > (XEN) r9: 0000000000000002 r10: ffff82d0802284f0 r11: > > > 0000000000000282 > > > (XEN) r12: 0000000000000000 r13: 00007fffd571aff0 r14: > > > ffff8300740a0000 > > > (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: > > > 00000000001526f0 > > > (XEN) cr3: 000000003afa0000 cr2: 000000007c6ebdb0 > > > (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 > > > (XEN) Xen stack trace from rsp=ffff82d0802b7dc0: > > > (XEN) ffff82d080103285 ffff83007c6eb000 0000000000000000 > ffff83007c6eb000 > > > (XEN) 0000000100000005 ffff83007c6eb000 ffff83007c6c1000 > > > ffff82d0802b7ec8 > > > (XEN) ffff82d0802b0000 ffff83007c6eb000 ffff82e000fde500 > 0000000000000005 > > > (XEN) 0000000000000000 ffff83007ef28000 ffff83007c6c1000 > 000000000007ef28 > > > (XEN) 00007ff000000003 0000000900000003 0000000001650001 > 000000000165f670 > > > (XEN) 00007fffd571b140 0000000000000001 00007f4b02299557 > 0000000000000001 > > > (XEN) 0000000000000000 00007fffd571b0ac 00007f4b01bf2018 > 000000000165f450 > > > (XEN) 0000000000000001 000000000165f680 00007f4b022a0515 > 0000000000000000 > > > (XEN) 0000000000000000 0000000000000000 0000000000000001 > 0000000000000033 > > > (XEN) ffff83007c6c1000 ffff8800442ebec0 ffff88006bdb2490 > > > 00007fffd571aff0 > > > (XEN) 00007fffd571aff0 0000000000000000 ffff82d08021aef9 > 00007fffd571b140 > > > (XEN) 000000000165f670 000000000165b050 000000000165f680 > ffff8800442ebec0 > > > (XEN) ffff88006d709400 0000000000000282 00007fff00000001 > 000000b4d327202c > > > (XEN) 00007f4b02077040 0000000000000024 ffffffff8100148a > 0000000000000000 > > > (XEN) 0000000000000001 00007f4b024a9004 0001010000000000 > ffffffff8100148a > > > (XEN) 000000000000e033 0000000000000282 ffff8800442ebe30 > 000000000000e02b > > > (XEN) 0000000000000000 0000000000000000 0000000000000000 > 0000000000000000 > > > (XEN) 0000000000000000 ffff83007c6c1000 0000000000000000 > 0000000000000000 > > > (XEN) Xen call trace: > > > (XEN) [<ffff82d080127b6b>] _spin_unlock+0x1b/0x30 > > > (XEN) [<ffff82d080103285>] do_domctl+0x2c5/0x1180 > > > (XEN) [<ffff82d08021aef9>] syscall_enter+0xa9/0xae > > > (XEN) > > > (XEN) Pagetable walk from 000000007c6ebdb0: > > > (XEN) L4[0x000] = 000000003afaf067 0000000000043139 > > > (XEN) L3[0x001] = 0000000000000000 ffffffffffffffff > > > (XEN) > > > (XEN) **************************************** > > > (XEN) Panic on CPU 0: > > > (XEN) FATAL PAGE FAULT > > > (XEN) [error_code=0002] > > > (XEN) Faulting linear address: 000000007c6ebdb0 > > > (XEN) **************************************** > > > (XEN) > > > (XEN) Reboot in five seconds... > > > > > > And the logs from L0 Xen (d20 is the L1 Xen): > > > (d20) HVM Loader > > > (d20) Detected Xen v4.6.0 > > > (d20) Xenbus rings @0xfeffc000, event channel 1 > > > (d20) System requested SeaBIOS > > > (d20) CPU speed is 1600 MHz > > > (d20) Relocating guest memory for lowmem MMIO space disabled > > > (d20) PCI-ISA link 0 routed to IRQ5 > > > (d20) PCI-ISA link 1 routed to IRQ10 > > > (d20) PCI-ISA link 2 routed to IRQ11 > > > (d20) PCI-ISA link 3 routed to IRQ5 > > > (d20) pci dev 01:3 INTA->IRQ10 > > > (d20) pci dev 02:0 INTA->IRQ11 > > > (d20) pci dev 04:0 INTA->IRQ5 > > > (d20) No RAM in high memory; setting high_mem resource base to 100000000 > > > (d20) pci dev 03:0 bar 10 size 002000000: 0f0000008 > > > (d20) pci dev 02:0 bar 14 size 001000000: 0f2000008 > > > (d20) pci dev 04:0 bar 30 size 000040000: 0f3000000 > > > (d20) pci dev 03:0 bar 30 size 000010000: 0f3040000 > > > (d20) pci dev 03:0 bar 14 size 000001000: 0f3050000 > > > (d20) pci dev 02:0 bar 10 size 000000100: 00000c001 > > > (d20) pci dev 04:0 bar 10 size 000000100: 00000c101 > > > (d20) pci dev 04:0 bar 14 size 000000100: 0f3051000 > > > (d20) pci dev 01:1 bar 20 size 000000010: 00000c201 > > > (d20) Multiprocessor initialisation: > > > (d20) - CPU0 ... 46-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... > > > done. > > > (d20) - CPU1 ... 46-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... > > > done. > > > (d20) Writing SMBIOS tables ... > > > (d20) Loading SeaBIOS ... > > > (d20) Creating MP tables ... > > > (d20) Loading ACPI ... > > > (d20) vm86 TSS at fc00a180 > > > (d20) BIOS map: > > > (d20) 10000-100d3: Scratch space > > > (d20) c0000-fffff: Main BIOS > > > (d20) E820 table: > > > (d20) [00]: 00000000:00000000 - 00000000:000a0000: RAM > > > (d20) HOLE: 00000000:000a0000 - 00000000:000c0000 > > > (d20) [01]: 00000000:000c0000 - 00000000:00100000: RESERVED > > > (d20) [02]: 00000000:00100000 - 00000000:7f800000: RAM > > > (d20) HOLE: 00000000:7f800000 - 00000000:fc000000 > > > (d20) [03]: 00000000:fc000000 - 00000001:00000000: RESERVED > > > (d20) Invoking SeaBIOS ... > > > (d20) SeaBIOS (version rel-1.8.2-0-g33fbe13-20151206_111754-storage) > > > (d20) > > > (d20) Found Xen hypervisor signature at 40000000 > > > (d20) Running on QEMU (i440fx) > > > (d20) xen: copy e820... > > > (d20) Relocating init from 0x000de2f0 to 0x7f7ae840 (size 71424) > > > (d20) CPU Mhz=1601 > > > (d20) Found 7 PCI devices (max PCI bus is 00) > > > (d20) Allocated Xen hypercall page at 7f7ff000 > > > (d20) Detected Xen v4.6.0 > > > (d20) xen: copy BIOS tables... > > > (d20) Copying SMBIOS entry point from 0x00010010 to 0x000f6490 > > > (d20) Copying MPTABLE from 0xfc001160/fc001170 to 0x000f6390 > > > (d20) Copying PIR from 0x00010030 to 0x000f6310 > > > (d20) Copying ACPI RSDP from 0x000100b0 to 0x000f62e0 > > > (d20) Using pmtimer, ioport 0xb008 > > > (d20) Scan for VGA option rom > > > (d20) Running option rom at c000:0003 > > > (d20) pmm call arg1=0 > > > (d20) Turning on vga text mode console > > > (d20) SeaBIOS (version rel-1.8.2-0-g33fbe13-20151206_111754-storage) > > > (d20) Machine UUID b99ffc8f-8dda-44c8-b8e7-04331274c410 > > > (d20) All threads complete. > > > (d20) Found 0 lpt ports > > > (d20) Found 1 serial ports > > > (d20) ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9) > > > (d20) ATA controller 2 at 170/374/0 (irq 15 dev 9) > > > (d20) ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (20480 MiBytes) > > > (d20) Searching bootorder for: /pci@i0cf8/*@1,1/drive@0/disk@0 > > > (d20) PS2 keyboard initialized > > > (d20) All threads complete. > > > (d20) Scan for option roms > > > (d20) Running option rom at c980:0003 > > > (d20) pmm call arg1=1 > > > (d20) pmm call arg1=0 > > > (d20) pmm call arg1=1 > > > (d20) pmm call arg1=0 > > > (d20) Searching bootorder for: /pci@i0cf8/*@4 > > > (d20) > > > (d20) Press F12 for boot menu. > > > (d20) > > > (d20) Searching bootorder for: HALT > > > (d20) drive 0x000f6290: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 > s=41943040 > > > (d20) Space available for UMB: ca800-ee800, f5cb0-f6290 > > > (d20) Returned 258048 bytes of ZoneHigh > > > (d20) e820 map has 6 items: > > > (d20) 0: 0000000000000000 - 000000000009fc00 = 1 RAM > > > (d20) 1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED > > > (d20) 2: 00000000000f0000 - 0000000000100000 = 2 RESERVED > > > (d20) 3: 0000000000100000 - 000000007f7ff000 = 1 RAM > > > (d20) 4: 000000007f7ff000 - 000000007f800000 = 2 RESERVED > > > (d20) 5: 00000000fc000000 - 0000000100000000 = 2 RESERVED > > > (d20) enter handle_19: > > > (d20) NULL > > > (d20) Booting from Hard Disk... > > > (d20) Booting from 0000:7c00 > > > > > > > > > jones > > > > > > From: quizy_jones@xxxxxxxxxxx > > > Date: 2015-12-11 09:42 > > > To: xen-devel > > > Subject: [Xen-devel] xen panics when setting int3 traps > > > I'd like to inject int3 traps to hypercall handlers to be able to capture > > > hypercalls. > However, the dom0/xen would reboot whenever I create a new VM. Followup is > the > console output when this happens. > > > > > > (XEN) Xen version 4.4.1 (Ubuntu 4.4.1-0ubuntu0.14.04.6) > (stefan.bader@xxxxxxxxxxxxx) (gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2) debug=n Wed > May > 20 12:19:20 UTC 2015 > > > (XEN) Bootloader: GRUB 2.02~beta2-9ubuntu1.2 > > > (XEN) Command line: placeholder console=com1 com1=115200 > dom0_max_vcpus=1 dom0_vcpus_pin > > > (XEN) Video information: > > > (XEN) VGA is text mode 80x25, font 8x16 > > > (XEN) Disc information: > > > (XEN) Found 1 MBR signatures > > > (XEN) Found 1 EDD information structures > > > (XEN) Xen-e820 RAM map: > > > (XEN) 0000000000000000 - 000000000009fc00 (usable) > > > (XEN) 000000000009fc00 - 00000000000a0000 (reserved) > > > (XEN) 00000000000f0000 - 0000000000100000 (reserved) > > > (XEN) 0000000000100000 - 000000007f7ff000 (usable) > > > (XEN) 000000007f7ff000 - 000000007f800000 (reserved) > > > (XEN) 00000000fc000000 - 0000000100000000 (reserved) > > > (XEN) System RAM: 2039MB (2088568kB) > > > (XEN) ACPI: RSDP 000F62E0, 0024 (r2 Xen) > > > (XEN) ACPI: XSDT FC00A090, 0054 (r1 Xen HVM 0 HVML > > > 0) > > > (XEN) ACPI: FACP FC0099C0, 00F4 (r4 Xen HVM 0 HVML > > > 0) > > > (XEN) ACPI: DSDT FC0012A0, 8691 (r2 Xen HVM 0 INTL > > > 20140214) > > > (XEN) ACPI: FACS FC001260, 0040 > > > (XEN) ACPI: APIC FC009AC0, 0460 (r2 Xen HVM 0 HVML > > > 0) > > > (XEN) ACPI: HPET FC009FA0, 0038 (r1 Xen HVM 0 HVML > > > 0) > > > (XEN) ACPI: WAET FC009FE0, 0028 (r1 Xen HVM 0 HVML > > > 0) > > > (XEN) ACPI: SSDT FC00A010, 0031 (r2 Xen HVM 0 INTL > > > 20140214) > > > (XEN) ACPI: SSDT FC00A050, 0031 (r2 Xen HVM 0 INTL > > > 20140214) > > > (XEN) Domain heap initialised > > > (XEN) Processor #0 7:15 APIC version 20 > > > (XEN) Processor #2 7:15 APIC version 20 > > > (XEN) IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-47 > > > (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs > > > (XEN) Not enabling x2APIC: depends on iommu_supports_eim. > > > (XEN) Using scheduler: SMP Credit Scheduler (credit) > > > (XEN) Detected 1600.055 MHz processor. > > > (XEN) Initing memory sharing. > > > (XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7 > > > (XEN) I/O virtualisation disabled > > > (XEN) ENABLING IO-APIC IRQs > > > (XEN) -> Using new ACK method > > > (XEN) Platform timer is 62.500MHz HPET > > > (XEN) Allocated console ring of 16 KiB. > > > (XEN) VMX: Supported advanced features: > > > (XEN) - APIC MMIO access virtualisation > > > (XEN) - APIC TPR shadow > > > (XEN) - Extended Page Tables (EPT) > > > (XEN) - Virtual-Processor Identifiers (VPID) > > > (XEN) - MSR direct-access bitmap > > > (XEN) - Unrestricted Guest > > > (XEN) HVM: ASIDs enabled. > > > (XEN) HVM: VMX enabled > > > (XEN) HVM: Hardware Assisted Paging (HAP) detected > > > (XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB > > > (XEN) Brought up 2 CPUs > > > (XEN) xenoprof: Initialization failed. Intel processor family 6 model > > > 63is not supported > > > (XEN) *** LOADING DOMAIN 0 *** > > > (XEN) Xen kernel: 64-bit, lsb, compat32 > > > (XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x23ff000 > > > (XEN) PHYSICAL MEMORY ARRANGEMENT: > > > (XEN) Dom0 alloc.: 0000000074000000->0000000078000000 (462231 pages to > be allocated) > > > (XEN) Init. ramdisk: 000000007def5000->000000007f1ff662 > > > (XEN) VIRTUAL MEMORY ARRANGEMENT: > > > (XEN) Loaded kernel: ffffffff81000000->ffffffff823ff000 > > > (XEN) Init. ramdisk: ffffffff823ff000->ffffffff83709662 > > > (XEN) Phys-Mach map: ffffffff8370a000->ffffffff83aba510 > > > (XEN) Start info: ffffffff83abb000->ffffffff83abb4b4 > > > (XEN) Page tables: ffffffff83abc000->ffffffff83add000 > > > (XEN) Boot stack: ffffffff83add000->ffffffff83ade000 > > > (XEN) TOTAL: ffffffff80000000->ffffffff83c00000 > > > (XEN) ENTRY ADDRESS: ffffffff81d341f0 > > > (XEN) Dom0 has maximum 1 VCPUs > > > (XEN) Scrubbing Free RAM: .done. > > > (XEN) Initial low memory virq threshold set at 0x4000 pages. > > > (XEN) Std. Loglevel: Errors and warnings > > > (XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings) > > > (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input > > > to Xen) > > > (XEN) Freed 272kB init memory. > > > mapping kernel into physical memory > > > about to get started... > > > --------------------------- logs when xen panics > > > (XEN) ----[ Xen-4.4.1 x86_64 debug=n Not tainted ]---- > > > (XEN) CPU: 0 > > > (XEN) RIP: e008:[<ffff82d080128132>] _write_unlock+0x22/0x40 > > > (XEN) RFLAGS: 0000000000010246 CONTEXT: hypervisor > > > (XEN) rax: 0000000080000000 rbx: ffff82e000233440 rcx: > > > 0000000000000000 > > > (XEN) rdx: 0000000000000000 rsi: 00000000000119a2 rdi: > > > ffff83007c6ca130 > > > (XEN) rbp: ffff83007c6eb000 rsp: ffff82d0802b7d40 r8: > > > 0000000000233440 > > > (XEN) r9: 0000000000000000 r10: ffff82d0802284f0 r11: > > > 0000000000000246 > > > (XEN) r12: 0000000000000001 r13: 00000000000119a2 r14: > > > 00000000000119a2 > > > (XEN) r15: ffff82d0802b7d64 cr0: 0000000080050033 cr4: > > > 00000000001526f0 > > > (XEN) cr3: 0000000075c0e000 cr2: 00007f54fd0cfe60 > > > (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 > > > (XEN) Xen stack trace from rsp=ffff82d0802b7d40: > > > (XEN) ffff82d080115b98 ffff82e000000001 ffff83007c6eb000 > 0000000000000001 > > > (XEN) 0000000000000000 ffff82e000eb81c0 0000000000000001 > 0000000000000000 > > > (XEN) ffff88006d62dd70 ffff82d0802b0000 ffff83007c6eb000 > 0000000000000000 > > > (XEN) ffff82d080116138 ffff83007c6eb000 ffff82e0002219c0 > 00000000000110ce > > > (XEN) ffff83007c6c1000 ffff83007c6eb000 0000000000000000 > 0000000000000200 > > > (XEN) ffffffff81fbf040 ffffffff00000001 ffff82d0802b0000 > > > ffff82d0802b7e70 > > > (XEN) ffff82d0802b7e80 ffff83007c6eb000 ffff83007c6c1000 > > > ffff82d0802b0000 > > > (XEN) ffff82d000000000 ffff83007c6eb000 ffff880000000000 > > > ffff83007c6c1000 > > > (XEN) 00007ff000000002 ffff83007c6eb000 0000000000000000 > ffff83007c6c1000 > > > (XEN) ffff82d08017c29c 00000021a5fe6264 00000000000119a2 > ffff82d0802f92c8 > > > (XEN) ffff82d0802b0000 ffff82d0802f8500 ffffffff81fbf040 > > > 0000000000000200 > > > (XEN) 0000000000000000 0000000000007ff0 0000000000000000 > ffff82d0802cf700 > > > (XEN) ffff82d0802b0000 ffffffffffffffff ffff83007c6c1000 > > > ffff88006d62ddb8 > > > (XEN) ffff83007c6c1000 ffff88006d62ddb8 0000000000000200 > 000000000006ba25 > > > (XEN) ffffea0000000000 ffffea0001ae8940 ffff82d08021aef9 > > > ffffea0001ae4c80 > > > (XEN) ffffea0000000000 000000000006b932 0000000000000200 > ffff88006d62ddb8 > > > (XEN) 0000000000000200 0000000000000246 0000000000007ff0 > 0000000000010bea > > > (XEN) 0000000000000000 000000000000000c ffffffff8100118a > 0000000000000000 > > > (XEN) ffff88006d62dd70 0000000000000001 0001010000000000 > ffffffff8100118a > > > (XEN) 000000000000e033 0000000000000246 ffff88006d62dd50 > 000000000000e02b > > > (XEN) Xen call trace: > > > (XEN) [<ffff82d080128132>] _write_unlock+0x22/0x40 > > > (XEN) [<ffff82d080115b98>] guest_remove_page+0xf8/0x2e0 > > > (XEN) [<ffff82d080116138>] do_memory_op+0x3b8/0x2750 > > > (XEN) [<ffff82d08017c29c>] __do_update_va_mapping+0x1bc/0x6c0 > > > (XEN) [<ffff82d08021aef9>] syscall_enter+0xa9/0xae > > > (XEN) > > > (XEN) > > > (XEN) **************************************** > > > (XEN) Panic on CPU 0: > > > (XEN) FATAL TRAP: vector = 6 (invalid opcode) > > > (XEN) **************************************** > > > (XEN) > > > (XEN) Reboot in five seconds... > > > > > > BTW, I'm using a nested xen 4.4.1 on xen 4.6.0. > > > > > > > > > Jones > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@xxxxxxxxxxxxx > > > http://lists.xen.org/xen-devel > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@xxxxxxxxxxxxx > > http://lists.xen.org/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |