[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] 4.2.1: Poor write performance for DomU.

On 9/03/2013 7:49 AM, Konrad Rzeszutek Wilk wrote:
On Fri, Mar 08, 2013 at 07:54:31PM +1100, Steven Haigh wrote:
On 20/02/2013 8:49 PM, Steven Haigh wrote:
On 20/02/2013 7:49 PM, Steven Haigh wrote:
On 20/02/2013 7:26 PM, Roger Pau Monné wrote:
On 20/02/13 03:10, Steven Haigh wrote:
Hi guys,

Firstly, please CC me in to any replies as I'm not a subscriber these

I've been trying to debug a problem with Xen 4.2.1 where I am unable to
achieve more than ~50Mb/sec sustained sequential write to a disk. The
DomU is configured as such:

Since you mention 4.2.1 explicitly, is this a performance regression
>from previous versions? (4.2.0 or the 4.1 branch)

This is actually a very good question. I've reinstalled my older
packages of Xen 4.1.3 back on the system. Rebooting into the new
hypervisor, then starting the single DomU again. Ran bonnie++ again on
the DomU:

Still around 50Mb/sec - so this doesn't seem to be a regression, but
something else?

I've actually done a bit of thinking about this... A recent thread on
linux-raid kernel mailing list about Xen and DomU throughput made me
revisit my setup. I know I used to be able to saturate GigE both ways
(send and receive) to the samba share served by this DomU. This would
mean I'd get at least 90-100Mbyte/sec. What exact config and kernel/xen
versions this was as this point in time I cannot say.

As such, I had a bit of a play and recreated my RAID6 with 64Kb chunk
size. This seemed to make rebuild/resync speeds way worse - so I
reverted to 128Kb chunk size.

The benchmarks I am getting from the Dom0 is about what I'd expect - but
I wouldn't expect to lose 130Mb/sec write speed to the phy:/ pass
through of the LV.

 From my known config where I could saturate the GigE connection, I have
changed from kernel 2.6.32 (Jeremy's git repo) to the latest vanilla
kernels - currently 3.7.9.

My build of Xen 4.2.1 also has all of the recent security advisories
patched as well. Although it is interesting to note that downgrading to
Xen 4.1.2 made no difference to write speeds.

Just wondering if there is any further news or tests that I might be
able to do on this?

So usually the problem like this is to unpeel the layers and find out
which of them is at fault. You have a stacked block system - LVM on
top of RAID6 on top of block devices.

To figure out who is interferring with the speeds I would recommend
you fault one of the RAID6 disks (so take it out of the RAID6). Pass
it to the guest as a raw disk (/dev/sdX as /dev/xvd) and then
run 'fio'. Run 'fio' as well in dom0 on the /dev/sdX and check
whether the write performance is different.

This is how I how do it:


Then progress up the stack. Try sticking the disk back in RAID6
and doing it on the RAID6. Then on the LVM and so on.

I did try to peel it back a single layer at a time. My test was simply using the same XFS filesystem in the Dom0 instead of the DomU.

I tested the underlying LVM config by mounting /dev/vg_raid6/fileshare from within the Dom0 and running bonnie++ as a benchmark:

Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP xenhost.lan.crc. 2G 667 96 186976 21 80430 14 956 95 290591 26 373.7 8
Latency             26416us     212ms     168ms   35494us   35989us 83759us
Version 1.96 ------Sequential Create------ --------Random Create-------- xenhost.lan.crc.id. -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 14901 32 +++++ +++ 19672 39 15307 34 +++++ +++ 18158 37
Latency             17838us     141us     298us     365us     133us 296us

~186Mb/sec write, ~290Mb/sec read. Awesome.

I then started a single DomU which gets passed /dev/vg_raid6/fileshare through as xvdb. It is then mounted in /mnt/fileshare/. I then ran bonnie++ again in the DomU:

Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP zeus.crc.id.au 2G 658 96 50618 8 42398 10 1138 99 267568 30 494.9 11
Latency             22959us     226ms     311ms   14617us   41816us 72814us
Version 1.96 ------Sequential Create------ --------Random Create-------- zeus.crc.id.au -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 21749 59 +++++ +++ 31089 73 23283 64 +++++ +++ 31114 75
Latency             18989us     164us     928us     480us      26us  87us

~50Mb/sec write, ~267Mb/sec read. Not so awesome.

As such, the filesystem, RAID6, etc are completely unchanged. The only change is the access method Dom0 vs DomU.

Steven Haigh

Email: netwiz@xxxxxxxxx
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature

Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.