Re: [Xen-devel] 4.2.1: Poor write performance for DomU.

[Steven Haigh <netwiz@xxxxxxxxx>]

On 12/03/2013 12:30 AM, Konrad Rzeszutek Wilk wrote:
> On Sat, Mar 09, 2013 at 09:30:54AM +1100, Steven Haigh wrote:
> > On 9/03/2013 7:49 AM, Konrad Rzeszutek Wilk wrote:
> > > On Fri, Mar 08, 2013 at 07:54:31PM +1100, Steven Haigh wrote:
> > > > On 20/02/2013 8:49 PM, Steven Haigh wrote:
> > > > > On 20/02/2013 7:49 PM, Steven Haigh wrote:
> > > > > > On 20/02/2013 7:26 PM, Roger Pau Monné wrote:
> > > > > > > On 20/02/13 03:10, Steven Haigh wrote:
> > > > > > > > Hi guys,
> > > > > > > >
> > > > > > > > Firstly, please CC me in to any replies as I'm not a subscriber these
> > > > > > > > days.
> > > > > > > >
> > > > > > > > I've been trying to debug a problem with Xen 4.2.1 where I am unable to
> > > > > > > > achieve more than ~50Mb/sec sustained sequential write to a disk. The
> > > > > > > > DomU is configured as such:
> > > > > > > Since you mention 4.2.1 explicitly, is this a performance regression
> > > > > > >from previous versions? (4.2.0 or the 4.1 branch)
> > > > > >
> > > > > > This is actually a very good question. I've reinstalled my older
> > > > > > packages of Xen 4.1.3 back on the system. Rebooting into the new
> > > > > > hypervisor, then starting the single DomU again. Ran bonnie++ again on
> > > > > > the DomU:
> > > > > >
> > > > > > Still around 50Mb/sec - so this doesn't seem to be a regression, but
> > > > > > something else?
> > > > > I've actually done a bit of thinking about this... A recent thread on
> > > > > linux-raid kernel mailing list about Xen and DomU throughput made me
> > > > > revisit my setup. I know I used to be able to saturate GigE both ways
> > > > > (send and receive) to the samba share served by this DomU. This would
> > > > > mean I'd get at least 90-100Mbyte/sec. What exact config and kernel/xen
> > > > > versions this was as this point in time I cannot say.
> > > > >
> > > > > As such, I had a bit of a play and recreated my RAID6 with 64Kb chunk
> > > > > size. This seemed to make rebuild/resync speeds way worse - so I
> > > > > reverted to 128Kb chunk size.
> > > > >
> > > > > The benchmarks I am getting from the Dom0 is about what I'd expect - but
> > > > > I wouldn't expect to lose 130Mb/sec write speed to the phy:/ pass
> > > > > through of the LV.
> > > > >
> > > > >  From my known config where I could saturate the GigE connection, I have
> > > > > changed from kernel 2.6.32 (Jeremy's git repo) to the latest vanilla
> > > > > kernels - currently 3.7.9.
> > > > >
> > > > > My build of Xen 4.2.1 also has all of the recent security advisories
> > > > > patched as well. Although it is interesting to note that downgrading to
> > > > > Xen 4.1.2 made no difference to write speeds.
> > > > Just wondering if there is any further news or tests that I might be
> > > > able to do on this?
> > > So usually the problem like this is to unpeel the layers and find out
> > > which of them is at fault. You have a stacked block system - LVM on
> > > top of RAID6 on top of block devices.
> > >
> > > To figure out who is interferring with the speeds I would recommend
> > > you fault one of the RAID6 disks (so take it out of the RAID6). Pass
> > > it to the guest as a raw disk (/dev/sdX as /dev/xvd) and then
> > > run 'fio'. Run 'fio' as well in dom0 on the /dev/sdX and check
> > > whether the write performance is different.
> > >
> > > This is how I how do it:
> > >
> > > [/dev/xvdXXX]
> > > rw=write
> > > direct=1
> > > size=4g
> > > ioengine=libaio
> > > iodepth=32
> > >
> > > Then progress up the stack. Try sticking the disk back in RAID6
> > > and doing it on the RAID6. Then on the LVM and so on.
> > I did try to peel it back a single layer at a time. My test was
> > simply using the same XFS filesystem in the Dom0 instead of the
> > DomU.
> Right, you are using a filesystem. That is another layer :-)
>
> And depending on what version of QEMU you have you might be using
> QEMU as the block PV backend instead of the kernel one. There
> were versions of QEMU that had highly inferior performance.
>
> Hence I was thinking of just using a raw disk to test that.
>
> > I tested the underlying LVM config by mounting
> > /dev/vg_raid6/fileshare from within the Dom0 and running bonnie++ as
> > a benchmark:
>
> So still filesystem. Fio can do it on a block level.
>
> What does 'xenstore-ls' show you and 'losetup -a'? I am really
> curious as to where that file you are providing to the guest as
> disk is being handled via 'loop' or via 'QEMU'.
I've picked out what I believe is the most relevant from xenstore-ls 
that belongs to the DomU in question:
     1 = ""
      51712 = ""
       domain = "zeus.vm"
       frontend = "/local/domain/1/device/vbd/51712"
       uuid = "3aa72be1-0e83-1ee2-a346-8ccef71e9d34"
       bootable = "1"
       dev = "xvda"
       state = "4"
       params = "/dev/RAID1/zeus.vm"
       mode = "w"
       online = "1"
       frontend-id = "1"
       type = "phy"
       physical-device = "fd:6"
       hotplug-status = "connected"
       feature-flush-cache = "1"
       feature-discard = "0"
       feature-barrier = "1"
       feature-persistent = "1"
       sectors = "135397376"
       info = "0"
       sector-size = "512"
      51728 = ""
       domain = "zeus.vm"
       frontend = "/local/domain/1/device/vbd/51728"
       uuid = "28375672-321c-0e33-4549-d64ee4daadec"
       bootable = "0"
       dev = "xvdb"
       state = "4"
       params = "/dev/vg_raid6/fileshare"
       mode = "w"
       online = "1"
       frontend-id = "1"
       type = "phy"
       physical-device = "fd:5"
       hotplug-status = "connected"
       feature-flush-cache = "1"
       feature-discard = "0"
       feature-barrier = "1"
       feature-persistent = "1"
       sectors = "5368709120"
       info = "0"
       sector-size = "512"

losetup -a returns nothing.


--
Steven Haigh

Email: netwiz@xxxxxxxxx
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel