Re: [Xen-devel] IO speed limited by size of IO request (for RBD driver)

[Felipe Franciosi <felipe.franciosi@xxxxxxxxxx>]

I noticed you copied your results from "dd", but I didn't see any conclusions 
drawn from experiment.

Did I understand it wrong or now you have comparable performance on dom0 and 
domU when using DIRECT?

domU:
# dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 25.4705 s, 84.3 MB/s

dom0:
# dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 24.8914 s, 86.3 MB/s


I think that if the performance differs when NOT using DIRECT, the issue must 
be related to the way your guest is flushing the cache. This must be generating 
a workload that doesn't perform well on Xen's PV protocol.

Cheers,
Felipe

-----Original Message-----
From: Steven Haigh [mailto:netwiz@xxxxxxxxx] 
Sent: 29 April 2013 20:48
To: Roger Pau Monne
Cc: Felipe Franciosi; xen-devel@xxxxxxxxxxxxx
Subject: Re: IO speed limited by size of IO request (for RBD driver)

On 30/04/2013 5:26 AM, Steven Haigh wrote:
> On 29/04/2013 6:38 PM, Roger Pau Monné wrote:
>> Did you also copy xen-blkfront?
>
> Dammit! No, no I didn't. I tried to just copy this back over to the
> 3.8.8 and 3.8.10 kernel versions, but it came up with too many errors 
> - so I just rebuilt/packages the checkout of your git based on 3.8.0-rc7.
>
>> It seems you are missing some pieces, you should see something like:
>>
>> blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; 
>> indirect descriptors: enabled;
>
> Now I'm running 3.8.0-rc7 from your git on both DomU and Dom0. In the 
> DomU, I now see:
>
> blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; 
> indirect descriptors: enabled;
>
>  From what you say, this should be what I'd expect.
>
>  From the DomU:
> # dd if=/dev/zero of=output.zero bs=1M count=2048
> 2048+0 records in
> 2048+0 records out
> 2147483648 bytes (2.1 GB) copied, 32.9252 s, 65.2 MB/s
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>             0.23    0.00    9.61    0.00    0.46   89.70
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz avgqu-sz   await  svctm  %util
> sdd            1071.40  7914.87   67.28  479.18     4.40    32.64
> 138.82    17.45   31.65   2.00 109.36
> sde            1151.72  7943.71   68.65  486.73     4.79    33.20
> 140.10    13.18   23.87   1.93 107.14
> sdc            1123.34  7921.05   66.36  482.84     4.66    32.86
> 139.89     8.80   15.96   1.86 102.31
> sdf            1091.53  7937.30   70.02  483.30     4.54    32.97
> 138.84    18.98   34.31   1.98 109.45
> md2               0.00     0.00    0.00 1003.66     0.00    65.31
> 133.27     0.00    0.00   0.00   0.00
>
> # dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct
> 2048+0 records in
> 2048+0 records out
> 2147483648 bytes (2.1 GB) copied, 25.4705 s, 84.3 MB/s
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>             0.22    0.00   10.94    0.00    0.22   88.62
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
> avgrq-sz avgqu-sz   await  svctm  %util
> sdd              20.35 13703.72    1.75  258.64     0.10    54.54
> 429.75     0.47    1.81   1.13  29.34
> sde            1858.64 11655.36   61.05  199.56     7.51    46.36
> 423.36     1.54    5.89   3.27  85.27
> sdc             142.45 11824.07    5.47  254.70     0.59    47.18
> 376.03     0.42    1.61   1.02  26.59
> sdf             332.39 13489.72   11.38  248.80     1.35    53.72
> 433.47     1.06    4.10   2.50  65.16
> md2               0.00     0.00    3.72  733.48     0.06    91.68
> 254.86     0.00    0.00   0.00   0.00

I just thought - I should probably include a baseline by mounting the same LV 
in the Dom0 and doing the exact same tests.

# dd if=/dev/zero of=output.zero bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 19.1554 s, 112 MB/s

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            0.00    0.00   23.18   76.60    0.22    0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sdd             139.07 14785.43   11.92  286.98     0.59    58.88 
407.50     2.60    8.71   1.84  54.92
sde              83.44 14846.58    8.39  292.05     0.36    59.09 
405.23     4.12   13.69   2.56  76.84
sdc              98.23 14828.04    9.93  289.18     0.42    58.84 
405.73     2.55    8.45   1.75  52.43
sdf              77.04 14816.78    8.61  289.40     0.33    58.96 
407.51     3.89   13.05   2.52  75.14
md2               0.00     0.00    0.00  973.51     0.00   116.72 
245.55     0.00    0.00   0.00   0.00

# dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 24.8914 s, 86.3 MB/s

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            0.00    0.00   12.22   87.58    0.21    0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sdd              32.09 12310.14    1.04  291.10     0.13    49.22 
345.99     0.48    1.66   0.91  26.71
sde            1225.88  9931.88   39.54  224.84     4.94    39.70 
345.81     1.20    4.53   2.44  64.55
sdc              19.25 11116.15    0.62  266.05     0.08    44.46 
342.06     0.41    1.53   0.86  22.94
sdf            1206.63 11122.77   38.92  253.21     4.87    44.51 
346.17     1.39    4.78   2.46  71.97
md2               0.00     0.00    0.00  634.37     0.00    79.30 
256.00     0.00    0.00   0.00   0.00

This is running the same kernel - 3.8.0-rc7 from your git.

And also for the sake completeness, the Dom0 grub.conf:
title Scientific Linux (3.8.0-1.el6xen.x86_64)
         root (hd0,0)
         kernel /xen.gz dom0_mem=1024M cpufreq=xen dom0_max_vcpus=1 
dom0_vcpus_pin
         module /vmlinuz-3.8.0-1.el6xen.x86_64 ro root=/dev/vg_raid1/xenhost 
rd_LVM_LV=vg_raid1/xenhost rd_MD_UUID=afb92c19:b9b1e3ae:07af315d:738e38be 
rd_NO_LUKS rd_NO_DM
LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us 
crashkernel=auto quiet panic=5 i915.i915_enable_rc6=7
i915.i915_enable_fbc=1 i915.lvds_downclock=1 drm.vblankoffdelay=1
         module /initramfs-3.8.0-1.el6xen.x86_64.img

and the DomU config:
# cat /etc/xen/zeus.vm
name            = "zeus.vm"
memory          = 1024
vcpus           = 2
cpus            = "1-3"
disk            = [ 'phy:/dev/vg_raid1/zeus.vm,xvda,w' , 
'phy:/dev/md2,xvdb,w' ]
vif             = [ "mac=02:16:36:35:35:09, bridge=br203, 
vifname=vm.zeus.203", "mac=10:16:36:35:35:09, bridge=br10, vifname=vm.zeus.10" ]
bootloader      = "pygrub"

on_poweroff     = 'destroy'
on_reboot       = 'restart'
on_crash        = 'restart'

All the tests are being done on /dev/md2 (from Dom0) presented as xvdb on the 
DomU.
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md2 : active raid6 sdd[5] sdc[4] sdf[1] sde[0]
       3906766592 blocks super 1.2 level 6, 128k chunk, algorithm 2 [4/4] [UUUU]

--
Steven Haigh

Email: netwiz@xxxxxxxxx
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel