Re: [Xen-devel] 4.2.1: Poor write performance for DomU.

[Steven Haigh <netwiz@xxxxxxxxx>]

On 9/03/2013 7:49 AM, Konrad Rzeszutek Wilk wrote:
> On Fri, Mar 08, 2013 at 07:54:31PM +1100, Steven Haigh wrote:
> > On 20/02/2013 8:49 PM, Steven Haigh wrote:
> > > On 20/02/2013 7:49 PM, Steven Haigh wrote:
> > > > On 20/02/2013 7:26 PM, Roger Pau Monné wrote:
> > > > > On 20/02/13 03:10, Steven Haigh wrote:
> > > > > > Hi guys,
> > > > > >
> > > > > > Firstly, please CC me in to any replies as I'm not a subscriber these
> > > > > > days.
> > > > > >
> > > > > > I've been trying to debug a problem with Xen 4.2.1 where I am unable to
> > > > > > achieve more than ~50Mb/sec sustained sequential write to a disk. The
> > > > > > DomU is configured as such:
> > > > > Since you mention 4.2.1 explicitly, is this a performance regression
> > > > >from previous versions? (4.2.0 or the 4.1 branch)
> > > >
> > > > This is actually a very good question. I've reinstalled my older
> > > > packages of Xen 4.1.3 back on the system. Rebooting into the new
> > > > hypervisor, then starting the single DomU again. Ran bonnie++ again on
> > > > the DomU:
> > > >
> > > > Still around 50Mb/sec - so this doesn't seem to be a regression, but
> > > > something else?
> > > I've actually done a bit of thinking about this... A recent thread on
> > > linux-raid kernel mailing list about Xen and DomU throughput made me
> > > revisit my setup. I know I used to be able to saturate GigE both ways
> > > (send and receive) to the samba share served by this DomU. This would
> > > mean I'd get at least 90-100Mbyte/sec. What exact config and kernel/xen
> > > versions this was as this point in time I cannot say.
> > >
> > > As such, I had a bit of a play and recreated my RAID6 with 64Kb chunk
> > > size. This seemed to make rebuild/resync speeds way worse - so I
> > > reverted to 128Kb chunk size.
> > >
> > > The benchmarks I am getting from the Dom0 is about what I'd expect - but
> > > I wouldn't expect to lose 130Mb/sec write speed to the phy:/ pass
> > > through of the LV.
> > >
> > >  From my known config where I could saturate the GigE connection, I have
> > > changed from kernel 2.6.32 (Jeremy's git repo) to the latest vanilla
> > > kernels - currently 3.7.9.
> > >
> > > My build of Xen 4.2.1 also has all of the recent security advisories
> > > patched as well. Although it is interesting to note that downgrading to
> > > Xen 4.1.2 made no difference to write speeds.
> > Just wondering if there is any further news or tests that I might be
> > able to do on this?
> So usually the problem like this is to unpeel the layers and find out
> which of them is at fault. You have a stacked block system - LVM on
> top of RAID6 on top of block devices.
>
> To figure out who is interferring with the speeds I would recommend
> you fault one of the RAID6 disks (so take it out of the RAID6). Pass
> it to the guest as a raw disk (/dev/sdX as /dev/xvd) and then
> run 'fio'. Run 'fio' as well in dom0 on the /dev/sdX and check
> whether the write performance is different.
>
> This is how I how do it:
>
> [/dev/xvdXXX]
> rw=write
> direct=1
> size=4g
> ioengine=libaio
> iodepth=32
>
> Then progress up the stack. Try sticking the disk back in RAID6
> and doing it on the RAID6. Then on the LVM and so on.
I did try to peel it back a single layer at a time. My test was simply 
using the same XFS filesystem in the Dom0 instead of the DomU.
I tested the underlying LVM config by mounting /dev/vg_raid6/fileshare 
from within the Dom0 and running bonnie++ as a benchmark:
Version  1.96       ------Sequential Output------ --Sequential Input- 
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP 
/sec %CP
xenhost.lan.crc. 2G   667  96 186976  21 80430  14   956  95 290591  26 
373.7   8
Latency             26416us     212ms     168ms   35494us   35989us 83759us
Version  1.96       ------Sequential Create------ --------Random 
Create--------
xenhost.lan.crc.id. -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP 
/sec %CP
                 16 14901  32 +++++ +++ 19672  39 15307  34 +++++ +++ 
18158  37
Latency             17838us     141us     298us     365us     133us 296us

~186Mb/sec write, ~290Mb/sec read. Awesome.

I then started a single DomU which gets passed /dev/vg_raid6/fileshare 
through as xvdb. It is then mounted in /mnt/fileshare/. I then ran 
bonnie++ again in the DomU:
Version  1.96       ------Sequential Output------ --Sequential Input- 
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP 
/sec %CP
zeus.crc.id.au   2G   658  96 50618   8 42398  10  1138  99 267568  30 
494.9  11
Latency             22959us     226ms     311ms   14617us   41816us 72814us
Version  1.96       ------Sequential Create------ --------Random 
Create--------
zeus.crc.id.au      -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP 
/sec %CP
                 16 21749  59 +++++ +++ 31089  73 23283  64 +++++ +++ 
31114  75
Latency             18989us     164us     928us     480us      26us  87us

~50Mb/sec write, ~267Mb/sec read. Not so awesome.

As such, the filesystem, RAID6, etc are completely unchanged. The only 
change is the access method Dom0 vs DomU.
--
Steven Haigh

Email: netwiz@xxxxxxxxx
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel