[Top] [All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] kernel bootup slow issue on ovm3.1.1



ä 2012-08-09 18:35, Jan Beulich åé: 
On 09.08.12 at 11:42, "zhenzhong.duan" <zhenzhong.duan@xxxxxxxxxx> wrote:

ä 2012-08-08 23:01, Jan Beulich åé:

On 08.08.12 at 11:48, "zhenzhong.duan"<zhenzhong.duan@xxxxxxxxxx>  wrote:

ä 2012-08-07 16:37, Jan Beulich åé:
Some spin at stop_machine after finish their job.

And here you'd need to find out what they're waiting for,
and what those CPUs are doing.

They are waiting the vcpu calling generic_set_all and those spin at 
set_atomicity_lock.
In fact, all are waiting generic_set_all

I think we're moving in circles - what is the vCPU currently
generic_set_all() then doing?
Add some debug print, generic_set_all->prepare_set->write_cr0 took much time,
all else are quick. set_atomicity_lock serialized this process between cpus, make it worse.
One iteration:
MTRR: CPU 2
prepare_set: before read_cr0
prepare_set: before write_cr0ÂÂÂ ------block here
prepare_set: before wbinvd
prepare_set: before read_cr4
prepare_set: before write_cr4
prepare_set: before __flush_tlb
prepare_set: before rdmsr
prepare_set: before wrmsr
generic_set_all: before set_mtrr_state
generic_set_all: before pat_init
post_set: before wbinvd
post_set: before wrmsr
post_set: before write_cr0
post_set: before write_cr4

 


There's not that much being done in generic_set_all(), so the
code should finish reasonably quickly. Are you perhaps having
more vCPU-s in the guest than pCPU-s they can run on?

System env is an exalogic node with 24 cores + 100G mem (2 socket , 6 
cores per socket, 2 HT threads per core).
Bootup a pvhvm with 12vpcus (or 24) + 90 GB + pci passthroughed device.

So you're indeed over-committing the system. How many vCPU-s
does you Dom0 have? Are there any other VMs? Is there any
vCPU pinning in effect?
dom0 boot with 24 vcpus(same result with dom0_max_vcpus=4). No other vm except dom0. All 24 vcpus spin from xentop result. Below is xentop clip.

ÂÂÂÂÂ NAMEÂ STATEÂÂ CPU(sec) CPU(%)ÂÂÂÂ MEM(k) MEM(%)Â MAXMEM(k) MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDSÂÂ VBD_OOÂÂ VBD_RDÂÂ VBD_WRÂ VBD_RSECTÂ VBD_WSECT
ÂSSID
 Domain-0 -----r 43072 158.8 2050560 2.0 no limit n/a 24 0 0 0 0 0 0 0 0 0
ÂÂÂ 0
VCPUs(sec): 0: 13649s 1: 6197s 2: 4254s 3: 2006s 4: 1409s
 5: 930s 6: 698s 7: 630s 8: 612s 9: 2038s
ÂÂÂÂÂÂÂÂ 10:ÂÂÂÂÂÂÂ 544s 11:ÂÂÂÂÂÂÂ 940s 12:ÂÂÂÂÂÂÂ 556s 13:ÂÂÂÂÂÂÂ 510s 14:ÂÂÂÂÂÂÂ 456s
ÂÂÂÂÂÂÂÂ 15:ÂÂÂÂÂÂÂ 591s 16:ÂÂÂÂÂÂÂ 438s 17:ÂÂÂÂÂÂÂ 508s 18:ÂÂÂÂÂÂ 3350s 19:ÂÂÂÂÂÂÂ 512s
ÂÂÂÂÂÂÂÂ 20:ÂÂÂÂÂÂÂ 544s 21:ÂÂÂÂÂÂÂ 529s 22:ÂÂÂÂÂÂÂ 547s 23:ÂÂÂÂÂÂÂ 610s
zduan_test -----rÂÂÂÂÂ 13140 2234.4ÂÂ 92327920ÂÂ 91.7ÂÂ 92327936ÂÂÂÂÂ 91.7ÂÂÂ 24ÂÂÂ 1ÂÂÂÂÂÂÂ 0ÂÂÂÂÂÂÂ 0ÂÂÂ 1ÂÂÂÂÂÂÂ 0ÂÂÂÂÂÂÂ 0ÂÂÂÂÂÂÂ 0ÂÂÂÂÂÂÂÂÂ 0ÂÂÂÂÂÂÂÂÂ 0
ÂÂÂ 0
VCPUs(sec): 0: 556s 1: 551s 2: 549s 3: 544s 4: 549s
 5: 545s 6: 545s 7: 547s 8: 545s 9: 548s
ÂÂÂÂÂÂÂÂ 10:ÂÂÂÂÂÂÂ 545s 11:ÂÂÂÂÂÂÂ 546s 12:ÂÂÂÂÂÂÂ 545s 13:ÂÂÂÂÂÂÂ 548s 14:ÂÂÂÂÂÂÂ 543s
ÂÂÂÂÂÂÂÂ 15:ÂÂÂÂÂÂÂ 544s 16:ÂÂÂÂÂÂÂ 551s 17:ÂÂÂÂÂÂÂ 545s 18:ÂÂÂÂÂÂÂ 547s 19:ÂÂÂÂÂÂÂ 551s
ÂÂÂÂÂÂÂÂ 20:ÂÂÂÂÂÂÂ 544s 21:ÂÂÂÂÂÂÂ 549s 22:ÂÂÂÂÂÂÂ 546s 23:ÂÂÂÂÂÂÂ 545s
  Does
your hardware support Pause-Loop-Exiting (or the AMD
equivalent, don't recall their term right now)?

I have no access to serial line, could I get the info by a command?

"xl dmesg" run early enough (i.e. before the log buffer wraps).
Below is xl dmesg result for your reference. thanks
[root@scae02cn01 zduan]# xl dmesg
Â__Â __ÂÂÂÂÂÂÂÂÂÂÂ _Â _ÂÂÂ ___ÂÂ ____ÂÂÂÂÂ _____ÂÂÂÂ ____Â __
Â\ \/ /___ _ __ÂÂ | || |Â / _ \ |___ \ÂÂÂ / _ \ \ÂÂ / /Â \/Â |
 \ // _ \ '_ \ | || |_| | | | __) |__| | | \ \ / /| |\/| |
 / \ __/ | | | |__ _| |_| | / __/|__| |_| |\ V / | | | |
Â/_/\_\___|_| |_|ÂÂÂ |_|(_)___(_)_____|ÂÂ \___/Â \_/Â |_|Â |_|

(XEN) Xen version 4.0.2-OVM (mockbuild@(none)) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) Fri Dec 23 17:00:16 EST 2011
(XEN) Latest ChangeSet: unavailable
(XEN) Bootloader: GNU GRUB 0.97
(XEN) Command line: dom0_mem=2G
(XEN) Video information:
(XEN)Â VGA is text mode 80x25, font 8x16
(XEN)Â VBE/DDC methods: none; EDID transfer time: 1 seconds
(XEN)Â EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)Â Found 1 MBR signatures
(XEN)Â Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)Â 0000000000000000 - 0000000000099400 (usable)
(XEN)Â 0000000000099400 - 00000000000a0000 (reserved)
(XEN)Â 00000000000e0000 - 0000000000100000 (reserved)
(XEN)Â 0000000000100000 - 000000007f780000 (usable)
(XEN)Â 000000007f78e000 - 000000007f790000 type 9
(XEN)Â 000000007f790000 - 000000007f79e000 (ACPI data)
(XEN)Â 000000007f79e000 - 000000007f7d0000 (ACPI NVS)
(XEN)Â 000000007f7d0000 - 000000007f7e0000 (reserved)
(XEN)Â 000000007f7ec000 - 0000000080000000 (reserved)
(XEN)Â 00000000e0000000 - 00000000f0000000 (reserved)
(XEN)Â 00000000fee00000 - 00000000fee01000 (reserved)
(XEN)Â 00000000ffc00000 - 0000000100000000 (reserved)
(XEN)Â 0000000100000000 - 0000001880000000 (usable)
(XEN) ACPI: RSDP 000FAA40, 0024 (r2 SUNÂÂ )
(XEN) ACPI: XSDT 7F790100, 0094 (r1 SUNÂÂÂ Xxx70ÂÂÂ 20111011 MSFTÂÂÂÂÂÂ 97)
(XEN) ACPI: FACP 7F790290, 00F4 (r4 SUNÂÂÂ Xxx70ÂÂÂ 20111011 MSFTÂÂÂÂÂÂ 97)
(XEN) ACPI: DSDT 7F7905C0, 5ECF (r2 SUNÂÂÂ Xxx70ÂÂÂÂÂÂÂÂÂÂ 1 INTL 20051117)
(XEN) ACPI: FACS 7F79E000, 0040
(XEN) ACPI: APIC 7F790390, 011E (r2 SUNÂÂÂ Xxx70ÂÂÂ 20111011 MSFTÂÂÂÂÂÂ 97)
(XEN) ACPI: MCFG 7F790500, 003C (r1 SUNÂÂÂ Xxx70ÂÂÂ 20111011 MSFTÂÂÂÂÂÂ 97)
(XEN) ACPI: SLIT 7F790540, 0030 (r1 SUNÂÂÂ Xxx70ÂÂÂ 20111011 MSFTÂÂÂÂÂÂ 97)
(XEN) ACPI: SPMI 7F790570, 0041 (r5 SUNÂÂÂ Xxx70ÂÂÂ 20111011 MSFTÂÂÂÂÂÂ 97)
(XEN) ACPI: OEMB 7F79E040, 00BE (r1 SUNÂÂÂ Xxx70ÂÂÂ 20111011 MSFTÂÂÂÂÂÂ 97)
(XEN) ACPI: HPET 7F79A5C0, 0038 (r1 SUNÂÂÂ Xxx70ÂÂÂ 20111011 MSFTÂÂÂÂÂÂ 97)
(XEN) ACPI: DMAR 7F79E100, 0130 (r1 SUNÂÂÂ Xxx70ÂÂÂÂÂÂÂÂÂÂ 1 MSFTÂÂÂÂÂÂ 97)
(XEN) ACPI: SRAT 7F79A600, 0250 (r1 SUNÂÂÂ Xxx70ÂÂÂÂÂÂÂÂÂÂ 1 INTCÂÂÂÂÂÂÂ 1)
(XEN) ACPI: SSDT 7F79EF60, 0363 (r1Â SUNÂÂ Xxx70ÂÂÂÂÂÂÂÂÂ 12 INTL 20051117)
(XEN) ACPI: EINJ 7F79A850, 0130 (r1 SUNÂÂÂ Xxx70ÂÂÂ 20111011 MSFTÂÂÂÂÂÂ 97)
(XEN) ACPI: BERT 7F79A9E0, 0030 (r1 SUNÂÂÂ Xxx70ÂÂÂ 20111011 MSFTÂÂÂÂÂÂ 97)
(XEN) ACPI: ERST 7F79AA10, 01B0 (r1 SUNÂÂÂ Xxx70ÂÂÂ 20111011 MSFTÂÂÂÂÂÂ 97)
(XEN) ACPI: HEST 7F79ABC0, 00A8 (r1 SUNÂÂÂ Xxx70ÂÂÂ 20111011 MSFTÂÂÂÂÂÂ 97)
(XEN) System RAM: 98295MB (100654180kB)
(XEN) Domain heap initialised DMA width 32 bits
(XEN) Processor #0 6:12 APIC version 21
(XEN) Processor #2 6:12 APIC version 21
(XEN) Processor #4 6:12 APIC version 21
(XEN) Processor #16 6:12 APIC version 21
(XEN) Processor #18 6:12 APIC version 21
(XEN) Processor #20 6:12 APIC version 21
(XEN) Processor #32 6:12 APIC version 21
(XEN) Processor #34 6:12 APIC version 21
(XEN) Processor #36 6:12 APIC version 21
(XEN) Processor #48 6:12 APIC version 21
(XEN) Processor #50 6:12 APIC version 21
(XEN) Processor #52 6:12 APIC version 21
(XEN) Processor #1 6:12 APIC version 21
(XEN) Processor #3 6:12 APIC version 21
(XEN) Processor #5 6:12 APIC version 21
(XEN) Processor #17 6:12 APIC version 21
(XEN) Processor #19 6:12 APIC version 21
(XEN) Processor #21 6:12 APIC version 21
(XEN) Processor #33 6:12 APIC version 21
(XEN) Processor #35 6:12 APIC version 21
(XEN) Processor #37 6:12 APIC version 21
(XEN) Processor #49 6:12 APIC version 21
(XEN) Processor #51 6:12 APIC version 21
(XEN) Processor #53 6:12 APIC version 21
(XEN) IOAPIC[0]: apic_id 6, version 32, address 0xfec00000, GSI 0-23
(XEN) IOAPIC[1]: apic_id 7, version 32, address 0xfec8a000, GSI 24-47
(XEN) Enabling APIC mode: Phys. Using 2 I/O APICs
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2926.029 MHz processor.
(XEN) Initing memory sharing.
(XEN) VMX: Supported advanced features:
(XEN)Â - APIC MMIO access virtualisation
(XEN)Â - APIC TPR shadow
(XEN)Â - Extended Page Tables (EPT)
(XEN)Â - Virtual-Processor Identifiers (VPID)
(XEN)Â - Virtual NMI
(XEN)Â - MSR direct-access bitmap
(XEN)Â - Unrestricted Guest
(XEN) EPT supports 2MB super page.
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging detected.
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) I/O virtualisation enabled
(XEN)Â - Dom0 mode: Relaxed
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) Total of 24 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)Â -> Using old ACK method
(XEN) TSC is reliable, synchronization unnecessary
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 64 KiB.
(XEN) Brought up 24 CPUs
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN)Â Dom0 kernel: 64-bit, lsb, paddr 0x2000 -> 0x6d5000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)Â Dom0 alloc.:ÂÂ 0000000835000000->0000000836000000 (520192 pages to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)Â Loaded kernel: ffffffff80002000->ffffffff806d5000
(XEN)Â Init. ramdisk: ffffffff806d5000->ffffffff80ed7400
(XEN)Â Phys-Mach map: ffffea0000000000->ffffea0000400000
(XEN)Â Start info:ÂÂÂ ffffffff80ed8000->ffffffff80ed84b4
(XEN)Â Page tables:ÂÂ ffffffff80ed9000->ffffffff80ee4000
(XEN)Â Boot stack:ÂÂÂ ffffffff80ee4000->ffffffff80ee5000
(XEN)Â TOTAL:ÂÂÂÂÂÂÂÂ ffffffff80000000->ffffffff81000000
(XEN)Â ENTRY ADDRESS: ffffffff80002000
(XEN) Dom0 has maximum 24 VCPUs
(XEN) Scrubbing Free RAM: .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 168kB init memory.

 
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel

 

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.

Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.