|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-users] Slow performance inside one VM
----- Original Message ----- > From: "Adam Goryachev" <mailinglists@xxxxxxxxxxxxxxxxxxxxxx> > To: "Xen-users" <Xen-users@xxxxxxxxxxxxx> > Sent: Friday, 6 April, 2018 03:33:03 > Subject: [Xen-users] Slow performance inside one VM > I have three windows VM's (domU's) running on the one physical box (dom0). > > 2 of them are Windows Server 2008R2, one of these is hardly used, but > the other is heavily used. > > 1 of them is Windows Server 2016, this one is reasonably heavily used > (number of users) but performs well. > > The VM with the performance problem has been happening for around 3 > months now, but I can't seem to find the issue. I've added additional > memory, but then saw that this wasn't used, so reduced it back a bit. > I've seen all CPU's (within the VM - taskmanager) pegged at 100% for > periods of time, so I've added additional vCPU's and this has stopped > that, but the "performance" issue is still present. > > Currently, the config file looks like this: > > builder = 'hvm' > memory = 20480 > name = "server1" > hostname = "server1" > localtime = 1 > vcpus = 22 # How many Virtual CPU's to present > viridian = 1 > disk = [ 'phy:/dev/ssd/server1,xvda,w' ] > boot = 'dc' > vif = ['bridge=xenbr0, mac=00:16:3e:43:ab:94'] > vnc = 1 > vncunused = 0 > vncdisplay = 5 > vnclisten = '0.0.0.0' > stdvga = 1 > #videoram = 16 > usb = 1 > usbdevice = 'tablet' > acpi = 1 > apic = 1 > on_reboot = 'restart' > on_poweroff = 'destroy' > on_crash = 'restart' > > The disk is a whole LV, using a RAID5 (Linux MD) from 3 x Intel SSD > > I suspect the issue is a disk issue based on Windows Resource Monitor > showing disk "Response Times" of over 30+ ms, up to 120ms I've seen. > > From the dom0 I can see one of the qemu-system-i386 processes is > sometimes over 50% cpu, but of the 4 CPU's allocated to dom0, none of > them are over 20% utilisation. > > 2133 ? RLsl 1117:45 /usr/bin/qemu-system-i386 -xen-domid 1 > -chardev socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-1,server,nowait > -no-shutdown -mon chardev=libxl-cmd,mode=control -chardev > socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-1,server,nowait > -mon chardev=libxenstat-cmd,mode=control -nodefaults -no-user-config > -name server3 -vnc 0.0.0.0:7 -display none -device > cirrus-vga,vgamem_mb=8 -boot order=cd -usb -usbdevice tablet -smp > 12,maxcpus=12 -device rtl8139,id=nic0,netdev=net0,mac=00:16:3e:3c:d7:16 > -netdev type=tap,id=net0,ifname=vif1.0-emu,script=no,downscript=no > -machine xenfv -m 24568 -drive > file=/dev/ssd/server3,if=ide,index=0,media=disk,format=raw,cache=writeback > > However, that process is from server3 (the win2016 domU) > > All software is installed from debian packages, versions are: > > dpkg -l | grep xen > ii grub-xen-bin 2.02~beta3-5 amd64 > GRand Unified Bootloader, version 2 (Xen binaries) > ii grub-xen-host 2.02~beta3-5 amd64 > GRand Unified Bootloader, version 2 (Xen host version) > ii libxen-4.4:amd64 4.4.1-9+deb8u10 > amd64 Public libs for Xen > ii libxen-4.8:amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 > amd64 Public libs for Xen > ii libxenstore3.0:amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 > amd64 Xenstore communications library for Xen > ii xen-hypervisor-4.4-amd64 4.4.1-9+deb8u10 > amd64 Xen Hypervisor on AMD64 > ii xen-hypervisor-4.8-amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 > amd64 Xen Hypervisor on AMD64 > ii xen-linux-system-3.16.0-4-amd64 > 3.16.51-3 amd64 Xen system with > Linux 3.16 on 64-bit PCs (meta-package) > ii xen-linux-system-3.16.0-5-amd64 > 3.16.51-3+deb8u1 amd64 Xen system with > Linux 3.16 on 64-bit PCs (meta-package) > ii xen-linux-system-amd64 4.9+80+deb9u4 > amd64 Xen system with Linux for 64-bit PCs (dummy package) > ii xen-system-amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 > amd64 Xen System on AMD64 (meta-package) > ii xen-utils-4.4 4.4.1-9+deb8u10 amd64 > XEN administrative tools > ii xen-utils-4.8 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 amd64 > XEN administrative tools > ii xen-utils-common 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 > all Xen administrative tools - common files > ii xenstore-utils 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 amd64 > Xenstore command line utilities for Xen > > xl info: > > host : xen3 > release : 4.9.0-6-amd64 > version : #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) > machine : x86_64 > nr_cpus : 40 > max_cpu_id : 39 > nr_nodes : 2 > cores_per_socket : 10 > threads_per_core : 2 > cpu_mhz : 2200 > hw_caps : > b7ebfbff:77fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100 > virt_caps : hvm hvm_directio > total_memory : 65403 > free_memory : 8841 > sharing_freed_memory : 0 > sharing_used_memory : 0 > outstanding_claims : 0 > free_cpus : 0 > xen_major : 4 > xen_minor : 8 > xen_extra : .3 > xen_version : 4.8.3 > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 > hvm-3.0-x86_32p hvm-3.0-x86_64 > xen_scheduler : credit > xen_pagesize : 4096 > platform_params : virt_start=0xffff800000000000 > xen_changeset : > xen_commandline : placeholder dom0_mem=6144M,max:6144M > dom0_max_vcpus=4 dom0_vcpus_pin dom0_mem=6144M,max:6144M > dom0_max_vcpus=4 dom0_vcpus_pin gnttab_max_frames=256 > cc_compiler : gcc (Debian 6.3.0-18) 6.3.0 20170516 > cc_compile_by : ijackson > cc_compile_domain : chiark.greenend.org.uk > cc_compile_date : Fri Mar 2 16:10:09 UTC 2018 > build_id : dff6bad5189f35adc717d7989e1e2c87b87860cc > xend_config_format : 4 > > Within the VM, I've installed the drivers > "Disk drive" displays as "XENSRC PVDISK SCSI Disk Device" > "Storage Controller" displays as "Xen PV Storage Host Adapter" where the > Driver shows Xen Project date 28/2/2017 version 8.2.0.9 > > Using iostat: /usr/bin/iostat -dmx /dev/sda /dev/sdb /dev/sdc /dev/md0 > /dev/md1 /dev/ssd/server1 /dev/ssd/server3 /dev/ssd/server2 1 > > Shows "% Util" is well under 20% (usually under 5%) > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz > avgqu-sz await r_await w_await svctm %util > sdc 1.00 18.00 2.00 47.00 0.01 0.21 > 9.41 0.01 0.16 0.00 0.17 0.16 0.80 > sda 0.00 18.00 2.00 37.00 0.01 0.17 > 9.56 0.01 0.31 0.00 0.32 0.31 1.20 > sdb 0.00 2.00 2.00 39.00 0.01 0.12 > 6.37 0.02 0.39 0.00 0.41 0.39 1.60 > md1 0.00 0.00 0.00 41.00 0.00 0.19 > 9.24 0.00 0.00 0.00 0.00 0.00 0.00 > md0 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > dm-6 0.00 0.00 0.00 14.00 0.00 0.04 > 6.50 0.02 1.14 0.00 1.14 0.57 0.80 > dm-10 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > dm-18 0.00 0.00 0.00 20.00 0.00 0.14 > 14.00 0.01 0.60 0.00 0.60 0.60 1.20 > > I'm not sure what other information I should provide, and to be honest, > I'm not sure what the bottleneck is. Users are seeing wait times of > greater than 10 seconds to open an application, or new email at times, > while other times it is near instant (less than 2s). All users are > connecting via RDP to this server. > > Any hints on how to prove that this is a disk performance issue, or CPU > or memory, or my suspicion, is some xen overhead/inefficiency issue > probably caused by misconfiguration on my part. The system did work > really well prior to January, but we did also add a number of additional > users to the system around that time. > > Thank you for any assistance or ideas. > > Regards, > Adam > You seem to have covered a lot of bases and I'm not sure what I can offer, especially without being at the server, but perhaps this issue I had will help you: https://serverfault.com/questions/660575/which-scheduler-to-change-on-lvm-to-benefit-virtual-machines Look at my own answer to the question. _______________________________________________ Xen-users mailing list Xen-users@xxxxxxxxxxxxxxxxxxxx https://lists.xenproject.org/mailman/listinfo/xen-users
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |