[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] Severe slow disk write issue on LVM-deployed DomU

Little info more:
on the Server with xen kernel 2.6.32-5, even in Dom0 I have slow disk write (10MB/s) on LVM volumes normally mounted :( I think the issue, on both server, is clearly related on the interaction between XEN and LVM2 layer

On 08/01/2013 12:16 PM, g wrote:
Hi Adam. Thanks for you quick answer. Am very happy to see that somebody
is involved in this issue. I feel less alone :)
Here below my reply, on line point by point:

On 08/01/2013 05:27 AM, Adam Goryachev wrote:
On 01/08/13 12:56, g wrote:
Hi everyone.
I have a really severe performance issue on disk write on my domU's PV
- Xen 4.1 fresh install on Debian Wheezy (kernel 3.2.0-4-amd64) via
- Simple disk (no raid) for Linux/Xen Dom0 installation
- LVM2 on top of raid5 disk raid5 disk for storing DomU
- Single LV for each DomU

The problem
While write speed (mesured via dd command) on Dom0 is near 163MB/s on
the mounted DomU LV , the same dd command provides 8MB/s if executed
the running DomU (with "noop" I/O scheduler set)
Running "top" in DomU, i  get high load average due to high %wa (no
high CPU load)

DomU disk is submitted to Xen via configuration file with:
disk  = [ 'phy:/dev/xenstore/test,xvda1,w' ]

If the same DomU is deployed on file image ("test.img" cloned from
"test" LV), write speed rise up to 63MB/s
disk  = [ 'file:/home/test.img,xvda1,w' ]

Obviously I can provide further detailed configuration parameters

Does anyone faced to the same problem? Does anyone can point me in the
right direction?

A couple of things to check.

shutdown the VM, and check the read/write speed on the LV from the dom0

The speed on LVM /dev/xenstore/test when DomU is powered off is
impressive: 163MB/s

Most likely, you have one or more snapshots of the LV, and this totally
kills performance. You might get away with a single snapshot and
performance might still be acceptable.

No snapshots active (never run any snapshot on this new Dom0 server)

Also, try adjusting the value of /sys/block/md1/md/stripe_cache_size
I ended up using 4096 which seemed to provide the best performance (I'm
using 4 x SSD in RAID5).

I'm not using software raid. Are you writing "md1" as generic block
device? In my case which block device should I adjust?

Verify performance on dom0 first, then compare it to the performance
within the domu.

Performance for disk write in Dom0 are great (> 20) if compared with
DomU deployed on LVM volume.
Disk write speed  on DomU deployed  on file-image disk are half the
speed of Dom0, but it's dangerous working with fileimage disk

Make sure you allocate sufficient CPU resources to dom0 as well, or
performance of the domu can suffer.

On this server (HP Proliant) I have 8GB RAM, 2x 8-core Intel CPUs, only
one domU with 1 vCpu, 1GB RAM, no swap
On another server (same model, old xen linux kernel 2.6.32-5) where the
problems began, I have 24GB RAM, 2x 8-core Intel CPUs, DomU with 4
vCPUs, 4GB swap

On both the server the problem is the same: 8MB/s disk write speed on
DomU if deployed on an LVM2 volume (unpartioned, with ext3 or ext4 FS)

Thanks to anyone can give help to this issue


Adam Goryachev Website Managers www.websitemanagers.com.au

Xen-users mailing list

Xen-users mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.