Re: [Xen-users] Xen 3.4.3 - bad IO performance on drbd devices

[Pasi Kärkkäinen <pasik@xxxxxx>]

On Fri, Jul 30, 2010 at 12:28:58PM +0200, Felix Botner wrote:
> Hi
> 
> some more information about the xen drbd performance problem (two servers, 
> each server has two drbd devices (protocol c, ext4) and is primary for one of 
> them). Any test without memory restriction for dom0. The network connection 
> between the drbd servers limits the troughput to ~230 MB for each drbd device 
> (2 x 1000 Mbps cards, bond mode 0). 
> 
> The local drive is a hardware raid10 with 12 300gb (3gbps 10k SAS) harddiscs. 
> The kernel is the latest debian/squeeze based kernel 2.6.32-5-amd64 (with 
> http://svn.debian.org/wsvn/kernel/?op=comp&compare[]=%2Fdists%2Fsid@15998&compare[]=%2Fdists%2Fsid@16001)
> 
> bonnie on the local drive with hypevisor
> ~ 400 MB/sec
> 
> bonnie on the connected drbd device with hypevisor
> ~ 170 MB/sec (bonnie on one drbd device/server)
> ~ 110 MB/sec (bonnie on both drbd devices/servers)
> 
> bonnie on the connected drbd device without hypevisor
> ~ 230 MB/sec (bonnie on one drbd device/server)
> ~ 200 MB/sec (bonnie on both drbd devices/servers)
> 
> bonnie on the disconnected drbd device with hypevisor
> ~ 300 MB/sec
> 
> bonnie on the disconnected drbd device without hypevisor
> ~ 360 MB/sec
> 
> What interests me is the throughput when writing to both servers. With the xen
> hypevisor i get 110 MB/sec on each drbd device (220 MB/sec io throughput on 
> each server because drbd writes "locale" and "remote"). Without the 
> hypervisor i get 200 MB/sec on each drbd device (400 MB/sec io throughput on 
> each server, the maximum of what the io backend allows).
> 
> But even with only one drbd device the io is without the hypevisor much 
> better 
> (230 MB/sec without and 170 MB/sec with hypervisor)
> 
> The strange thing is, a drbd resync gives me with or without hypervisor ~ 230 
> MB/sec. And when i start one server without hypervisor, bonnie gives me ~ 230 
> MB/sec on that server and on the remote server with the hypervisor (because 
> drbd also writes remote).
> 
> Any hints?
> 

I guess DRBD performance mainly depends on the latency you have,
because of the synchronization needed between the hosts.

Have you tried pinning the dom0 vcpus?

-- Pasi


> Thanks,
> Felix
> 
> xm info
> host                   : samla
> release                : 2.6.32-ucs9-xen-amd64
> version                : #1 SMP Thu Jul 22 04:32:22 UTC 2010
> machine                : x86_64
> nr_cpus                : 8
> nr_nodes               : 1
> cores_per_socket       : 4
> threads_per_core       : 1
> cpu_mhz                : 2133
> hw_caps                : 
> bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000
> virt_caps              : hvm
> total_memory           : 24564
> free_memory            : 22230
> node_to_cpu            : node0:0-7
> node_to_memory         : node0:22230
> xen_major              : 3
> xen_minor              : 4
> xen_extra              : .3
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          : unavailable
> cc_compiler            : gcc version 4.3.2 (Debian 4.3.2-1.1.13.200909082302)
> cc_compile_by          : root
> cc_compile_domain      : [unknown]
> cc_compile_date        : Thu Jun 17 14:54:34 UTC 2010
> xend_config_format     : 4
> 
> Am Dienstag 20 Juli 2010 15:54:21 schrieb Felix Botner:
> > Am Dienstag 20 Juli 2010 14:35:30 schrieb Pasi Kärkkäinen:
> > > On Tue, Jul 20, 2010 at 10:46:34AM +0200, Felix Botner wrote:
> > > > Hi everyone,
> > > >
> > > > i have two servers installed with a debian/lenny based os (64bit), a
> > > > debian/sid based kernel 2.6.32-xen-amd64 and xen 3.4.3-4. Each server
> > > > has two drbd devices (protocol c, formatted with ext4) and is primary
> > > > for one of them. Every drbd pair has a dedicated network interface (a
> > > > bond mode 0 interface with two 1000 Mbps cards).
> > >
> > > <snip>
> > >
> > > > The io performance on the connected drbd devices is significantly worse
> > > > if i start the kernel with the xen hypervisor (with "kernel
> > > > /boot/xen-3.4.3.gz"). Without the hypervisor (but the same kernel) the
> > > > systems are about 50% faster.
> > >
> > > Are you measuring from dom0 or from a guest?
> >
> > From dom0, there are no guests at the moment.
> >
> > > > Why is there such a difference?
> > > > Can i optimize my xend (i already added dom0_mem=2048M dom0_max_vcpus=2
> > > > dom0_vcpus_pin as boot parameter with no effect)?
> > > > Are there any known issues using XEN and bonding/drbd?
> > > >
> > > > Feel free to ask for more information about the system or the setup.
> > >
> > > How much memory does your server have?
> > > ie. how much ram do you have when you boot it baremetal without Xen.
> >
> > ~20GB without Xen. Now i removed the hypervisor param dom0_mem=2048M from
> > menu.lst (in xend-config set (dom0-min-mem 196) and (enable-dom0-ballooning
> > yes)), rebooted and as far as i know there should be no memory restriction
> > for dom0. "xm list" shows me the complete memory for dom0:
> >
> > server1-> xm list
> > Name                                        ID   Mem VCPUs      State
> > Time(s)
> > Domain-0                                     0 18323     2     r-----  
> > 1785.0
> >
> > server2-> xm list
> > Name                                        ID   Mem VCPUs      State
> > Time(s)
> > Domain-0                                     0 22375     2     r-----  
> > 1754.2
> >
> > But bonnie++ gives me still bad results:
> >
> > server1,60000M,,,113444,30,55569,13,,,231555,23,622.6,0,16,25634,70,+++++,+
> >++, +++++,+++,+++++,+++,+++++,+++,+++++,+++
> > server2,60000M,,,114014,31,53864,13,,,243541,27,617.4,0,16,+++++,+++,+++++,
> > +++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
> >
> > So i don't think caching is the issue, or?
> >
> > bye
> >
> > Felix
> >
> > > Remember all the free memory of the host/dom0 will be used by linux
> > > pagecache.. So if you limit dom0 to 2GB, it'll have less cache than the
> > > baremetal case.
> > >
> > > -- Pasi
> 
> 
> -- 
> Felix Botner
> 
> Open Source Software Engineer
> 
> Univention GmbH
> Linux for your business
> Mary-Somerville-Str.1
> 28359 Bremen
> Tel. : +49 421 22232-0
> Fax : +49 421 22232-99
> 
> botner@xxxxxxxxxxxxx>
> http://www.univention.de
> 
> Geschäftsführer: Peter H. Ganten
> HRB 20755 Amtsgericht Bremen
> Steuer-Nr.: 71-597-02876 
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users