[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] 4.2.1: Poor write performance for DomU.

On Sat, Mar 09, 2013 at 09:30:54AM +1100, Steven Haigh wrote:
> On 9/03/2013 7:49 AM, Konrad Rzeszutek Wilk wrote:
> >On Fri, Mar 08, 2013 at 07:54:31PM +1100, Steven Haigh wrote:
> >>On 20/02/2013 8:49 PM, Steven Haigh wrote:
> >>>On 20/02/2013 7:49 PM, Steven Haigh wrote:
> >>>>On 20/02/2013 7:26 PM, Roger Pau Monné wrote:
> >>>>>On 20/02/13 03:10, Steven Haigh wrote:
> >>>>>>Hi guys,
> >>>>>>
> >>>>>>Firstly, please CC me in to any replies as I'm not a subscriber these
> >>>>>>days.
> >>>>>>
> >>>>>>I've been trying to debug a problem with Xen 4.2.1 where I am unable to
> >>>>>>achieve more than ~50Mb/sec sustained sequential write to a disk. The
> >>>>>>DomU is configured as such:
> >>>>>
> >>>>>Since you mention 4.2.1 explicitly, is this a performance regression
> >>>>>from previous versions? (4.2.0 or the 4.1 branch)
> >>>>
> >>>>This is actually a very good question. I've reinstalled my older
> >>>>packages of Xen 4.1.3 back on the system. Rebooting into the new
> >>>>hypervisor, then starting the single DomU again. Ran bonnie++ again on
> >>>>the DomU:
> >>>>
> >>>>Still around 50Mb/sec - so this doesn't seem to be a regression, but
> >>>>something else?
> >>>
> >>>I've actually done a bit of thinking about this... A recent thread on
> >>>linux-raid kernel mailing list about Xen and DomU throughput made me
> >>>revisit my setup. I know I used to be able to saturate GigE both ways
> >>>(send and receive) to the samba share served by this DomU. This would
> >>>mean I'd get at least 90-100Mbyte/sec. What exact config and kernel/xen
> >>>versions this was as this point in time I cannot say.
> >>>
> >>>As such, I had a bit of a play and recreated my RAID6 with 64Kb chunk
> >>>size. This seemed to make rebuild/resync speeds way worse - so I
> >>>reverted to 128Kb chunk size.
> >>>
> >>>The benchmarks I am getting from the Dom0 is about what I'd expect - but
> >>>I wouldn't expect to lose 130Mb/sec write speed to the phy:/ pass
> >>>through of the LV.
> >>>
> >>> From my known config where I could saturate the GigE connection, I have
> >>>changed from kernel 2.6.32 (Jeremy's git repo) to the latest vanilla
> >>>kernels - currently 3.7.9.
> >>>
> >>>My build of Xen 4.2.1 also has all of the recent security advisories
> >>>patched as well. Although it is interesting to note that downgrading to
> >>>Xen 4.1.2 made no difference to write speeds.
> >>>
> >>
> >>Just wondering if there is any further news or tests that I might be
> >>able to do on this?
> >
> >So usually the problem like this is to unpeel the layers and find out
> >which of them is at fault. You have a stacked block system - LVM on
> >top of RAID6 on top of block devices.
> >
> >To figure out who is interferring with the speeds I would recommend
> >you fault one of the RAID6 disks (so take it out of the RAID6). Pass
> >it to the guest as a raw disk (/dev/sdX as /dev/xvd) and then
> >run 'fio'. Run 'fio' as well in dom0 on the /dev/sdX and check
> >whether the write performance is different.
> >
> >This is how I how do it:
> >
> >[/dev/xvdXXX]
> >rw=write
> >direct=1
> >size=4g
> >ioengine=libaio
> >iodepth=32
> >
> >Then progress up the stack. Try sticking the disk back in RAID6
> >and doing it on the RAID6. Then on the LVM and so on.
> I did try to peel it back a single layer at a time. My test was
> simply using the same XFS filesystem in the Dom0 instead of the
> DomU.

Right, you are using a filesystem. That is another layer :-)

And depending on what version of QEMU you have you might be using
QEMU as the block PV backend instead of the kernel one. There
were versions of QEMU that had highly inferior performance.

Hence I was thinking of just using a raw disk to test that.

> I tested the underlying LVM config by mounting
> /dev/vg_raid6/fileshare from within the Dom0 and running bonnie++ as
> a benchmark:

So still filesystem. Fio can do it on a block level.

What does 'xenstore-ls' show you and 'losetup -a'? I am really
curious as to where that file you are providing to the guest as
disk is being handled via 'loop' or via 'QEMU'.

Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.