[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] I/O Performance concerns (Xen 3.3.2)


  • To: xen-users@xxxxxxxxxxxxxxxxxxx
  • From: Dot Yet <dot.yet@xxxxxxxxx>
  • Date: Fri, 30 Oct 2009 00:59:55 -0400
  • Delivery-date: Thu, 29 Oct 2009 22:00:51 -0700
  • Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=D54LLm5IITdeAwrWRmLVEtCluVSxRqo1uJ5ldINFyaGmyP7Ovx3Epgianc6oiX8feu /d/lNj4xEgQ+jBzmt6DvDXjPiVpU5ni4d5BogdZAZkjTjeBGEtMK9Ly9ifwaLgbaEkyq HX8kGsQ/f4/vjeP1p1XyE/qV9ymegpWJQVdvQ=
  • List-id: Xen user discussion <xen-users.lists.xensource.com>

Hi everyone,

I need some help with regard to I/O performance on a xen based environment. Following is the setup.

Host Server Configuration:

CPU: 2 x 5430 Xeon CPUs
Motherboard: Tempest i5400PW S5397
RAM: 24GB RAM
Disks: 8 x 500 GB 7200 RPM disks - for VMs
SATA controller on 4 disks: Supermicro 8-Port SATA Card - (AOC-SAT2-MV8) 
SATA controller on other 4 disks: onboard controller (Intel 6321ESB)
1 x 250 GB 7200 RPM disk - for OS
OS: OpenSolaris 2009.06 with Xen 3.3.2

The relevant zfs pool 'vmdisk' has been laid out as follows:

               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
vmdisk       188G  1.18T      0      0      0      0
  mirror    62.7G   401G      0      0      0      0
    c8t2d0      -      -      0      0      0      0
    c9t0d0      -      -      0      0      0      0
  mirror    62.7G   401G      0      0      0      0
    c8t3d0      -      -      0      0      0      0
    c9t1d0      -      -      0      0      0      0
  mirror    62.7G   401G      0      0      0      0
    c8t4d0      -      -      0      0      0      0
    c9t4d0      -      -      0      0      0      0
  c8t5d0        0   464G      0      0      0      0
cache           -      -      -      -      -      -
  c9t5d0    76.6G   389G      0      0      0      0
----------  -----  -----  -----  -----  -----  -----

Where c8t5d0 is a slog device and c9t5d0 is a cache device. The recordsize is default 128k and compression is set to OFF.

Xen dom0 has been limited to 2 GB RAM through following configuration in /rpool/boot/grub/menu.lst file:

kernel$ /boot/$ISADIR/xen.gz dom0_mem=2048M

domU has been configured as follows:

VCPUs - 4
RAM - 8GB
HDD - 100GB zvol on ZFS filesystem 'vmdisk'. The record size of the zvol is default of 8k and compression is set to OFF. Also used --nonsparse command line option while creating the VM through virt-install.
Paravirtualized
OS: CentOS release 5.4 (Final) X86_64
Kernel: 2.6.18-164.el5xen


Now, the problem. I am running the following dd command inside the domU:

[user1@db2db01 ~]$ dd if=/dev/zero of=j1 bs=8k count=100000
100000+0 records in
100000+0 records out
819200000 bytes (819 MB) copied, 2.57249 seconds, 318 MB/s


The above command returns under 2.5 seconds, however iostat on domU AND zpool iostat on dom0, both continue to show write IO activity for upto 30 to 40 seconds:

domU iostat:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    0.00   24.81    0.00   75.19

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
xvda              0.00  4989.00  0.00 1133.00     0.00    23.06    41.68   146.08  107.49   0.88 100.00
xvda1             0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
xvda2             0.00  4989.00  0.00 1133.00     0.00    23.06    41.68   146.08  107.49   0.88 100.00
dm-0              0.00     0.00  0.00 6113.00     0.00    23.88     8.00   759.28  100.24   0.16 100.00
dm-1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    0.00   24.94    0.00   75.06

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
xvda              0.00  4989.00  0.00 1153.00     0.00    23.91    42.47   146.32  146.37   0.87 100.40
xvda1             0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
xvda2             0.00  4989.00  0.00 1153.00     0.00    23.91    42.47   146.32  146.37   0.87 100.40
dm-0              0.00     0.00  0.00 6143.00     0.00    24.00     8.00   751.75  143.43   0.16 100.40
dm-1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

dom0 zpool iostat:
               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
vmdisk       188G  1.18T    287  2.57K  2.24M  19.7M
  mirror    62.7G   401G    142    890  1.12M  6.65M
    c8t2d0      -      -     66    352   530K  6.49M
    c9t0d0      -      -     76    302   612K  6.65M
  mirror    62.7G   401G     83    856   670K  6.39M
    c8t3d0      -      -     43    307   345K  6.40M
    c9t1d0      -      -     40    293   325K  6.40M
  mirror    62.7G   401G     60    886   485K  6.68M
    c8t4d0      -      -     50    373   402K  6.68M
    c9t4d0      -      -     10    307  82.9K  6.68M
  c8t5d0        0   464G      0      0      0      0
cache           -      -      -      -      -      -
  c9t5d0    77.1G   389G    472     38  3.86M  3.50M
----------  -----  -----  -----  -----  -----  -----

               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
vmdisk       188G  1.18T     75  3.52K   594K  27.1M
  mirror    62.7G   401G     30  1.16K   239K  8.89M
    c8t2d0      -      -     10    464  86.6K  8.89M
    c9t0d0      -      -     19    350   209K  8.89M
  mirror    62.7G   401G      0  1.18K      0  9.10M
    c8t3d0      -      -      0    510      0  9.10M
    c9t1d0      -      -      0    385      0  9.10M
  mirror    62.7G   401G     45  1.18K   355K  9.11M
    c8t4d0      -      -     37    469   354K  9.11M
    c9t4d0      -      -      7    391  57.7K  9.11M
  c8t5d0        0   464G      0      0      0      0
cache           -      -      -      -      -      -
  c9t5d0    77.1G   389G    514    157  4.14M  17.4M
----------  -----  -----  -----  -----  -----  -----

Can you tell me why this happens? Is this behavior coming from Linux or Xen or ZFS? I do notice that iostat reports an iowait of 25%, but I don't know who is causing this bottleneck amongst them. 

Instead of writing a 800 meg file, if I write an 8gb file, the performance is very poor (40 MB/s or so) and again, despite there is a long iowait after the dd command returns.

Any help would be really appreciated. 

Kind regards,
dot.yet
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.