Re: [Xen-devel] IO speed limited by size of IO request (for RBD driver)

[Steven Haigh <netwiz@xxxxxxxxx>]

On 30/04/2013 5:26 AM, Steven Haigh wrote:
> On 29/04/2013 6:38 PM, Roger Pau Monné wrote:
> > Did you also copy xen-blkfront?
> Dammit! No, no I didn't. I tried to just copy this back over to the
> 3.8.8 and 3.8.10 kernel versions, but it came up with too many errors -
> so I just rebuilt/packages the checkout of your git based on 3.8.0-rc7.
>
> > It seems you are missing some pieces, you should see something like:
> >
> > blkfront: xvda: flush diskcache: enabled; persistent grants: enabled;
> > indirect descriptors: enabled;
> Now I'm running 3.8.0-rc7 from your git on both DomU and Dom0. In the
> DomU, I now see:
>
> blkfront: xvda: flush diskcache: enabled; persistent grants: enabled;
> indirect descriptors: enabled;
>
>  From what you say, this should be what I'd expect.
>
>  From the DomU:
> # dd if=/dev/zero of=output.zero bs=1M count=2048
> 2048+0 records in
> 2048+0 records out
> 2147483648 bytes (2.1 GB) copied, 32.9252 s, 65.2 MB/s
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>             0.23    0.00    9.61    0.00    0.46   89.70
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz avgqu-sz   await  svctm  %util
> sdd            1071.40  7914.87   67.28  479.18     4.40    32.64
> 138.82    17.45   31.65   2.00 109.36
> sde            1151.72  7943.71   68.65  486.73     4.79    33.20
> 140.10    13.18   23.87   1.93 107.14
> sdc            1123.34  7921.05   66.36  482.84     4.66    32.86
> 139.89     8.80   15.96   1.86 102.31
> sdf            1091.53  7937.30   70.02  483.30     4.54    32.97
> 138.84    18.98   34.31   1.98 109.45
> md2               0.00     0.00    0.00 1003.66     0.00    65.31
> 133.27     0.00    0.00   0.00   0.00
>
> # dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct
> 2048+0 records in
> 2048+0 records out
> 2147483648 bytes (2.1 GB) copied, 25.4705 s, 84.3 MB/s
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>             0.22    0.00   10.94    0.00    0.22   88.62
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz avgqu-sz   await  svctm  %util
> sdd              20.35 13703.72    1.75  258.64     0.10    54.54
> 429.75     0.47    1.81   1.13  29.34
> sde            1858.64 11655.36   61.05  199.56     7.51    46.36
> 423.36     1.54    5.89   3.27  85.27
> sdc             142.45 11824.07    5.47  254.70     0.59    47.18
> 376.03     0.42    1.61   1.02  26.59
> sdf             332.39 13489.72   11.38  248.80     1.35    53.72
> 433.47     1.06    4.10   2.50  65.16
> md2               0.00     0.00    3.72  733.48     0.06    91.68
> 254.86     0.00    0.00   0.00   0.00
I just thought - I should probably include a baseline by mounting the 
same LV in the Dom0 and doing the exact same tests.
# dd if=/dev/zero of=output.zero bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 19.1554 s, 112 MB/s

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00   23.18   76.60    0.22    0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sdd             139.07 14785.43   11.92  286.98     0.59    58.88 
407.50     2.60    8.71   1.84  54.92
sde              83.44 14846.58    8.39  292.05     0.36    59.09 
405.23     4.12   13.69   2.56  76.84
sdc              98.23 14828.04    9.93  289.18     0.42    58.84 
405.73     2.55    8.45   1.75  52.43
sdf              77.04 14816.78    8.61  289.40     0.33    58.96 
407.51     3.89   13.05   2.52  75.14
md2               0.00     0.00    0.00  973.51     0.00   116.72 
245.55     0.00    0.00   0.00   0.00
# dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 24.8914 s, 86.3 MB/s

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00   12.22   87.58    0.21    0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sdd              32.09 12310.14    1.04  291.10     0.13    49.22 
345.99     0.48    1.66   0.91  26.71
sde            1225.88  9931.88   39.54  224.84     4.94    39.70 
345.81     1.20    4.53   2.44  64.55
sdc              19.25 11116.15    0.62  266.05     0.08    44.46 
342.06     0.41    1.53   0.86  22.94
sdf            1206.63 11122.77   38.92  253.21     4.87    44.51 
346.17     1.39    4.78   2.46  71.97
md2               0.00     0.00    0.00  634.37     0.00    79.30 
256.00     0.00    0.00   0.00   0.00
This is running the same kernel - 3.8.0-rc7 from your git.

And also for the sake completeness, the Dom0 grub.conf:
title Scientific Linux (3.8.0-1.el6xen.x86_64)
        root (hd0,0)
        kernel /xen.gz dom0_mem=1024M cpufreq=xen dom0_max_vcpus=1 
dom0_vcpus_pin
        module /vmlinuz-3.8.0-1.el6xen.x86_64 ro 
root=/dev/vg_raid1/xenhost rd_LVM_LV=vg_raid1/xenhost 
rd_MD_UUID=afb92c19:b9b1e3ae:07af315d:738e38be rd_NO_LUKS rd_NO_DM 
LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us 
crashkernel=auto quiet panic=5 i915.i915_enable_rc6=7 
i915.i915_enable_fbc=1 i915.lvds_downclock=1 drm.vblankoffdelay=1
        module /initramfs-3.8.0-1.el6xen.x86_64.img

and the DomU config:
# cat /etc/xen/zeus.vm
name            = "zeus.vm"
memory          = 1024
vcpus           = 2
cpus            = "1-3"
disk            = [ 'phy:/dev/vg_raid1/zeus.vm,xvda,w' , 
'phy:/dev/md2,xvdb,w' ]
vif             = [ "mac=02:16:36:35:35:09, bridge=br203, 
vifname=vm.zeus.203", "mac=10:16:36:35:35:09, bridge=br10, 
vifname=vm.zeus.10" ]
bootloader      = "pygrub"

on_poweroff     = 'destroy'
on_reboot       = 'restart'
on_crash        = 'restart'

All the tests are being done on /dev/md2 (from Dom0) presented as xvdb 
on the DomU.
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md2 : active raid6 sdd[5] sdc[4] sdf[1] sde[0]
      3906766592 blocks super 1.2 level 6, 128k chunk, algorithm 2 
[4/4] [UUUU]
--
Steven Haigh

Email: netwiz@xxxxxxxxx
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel