[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] IO speed limited by size of IO request (for RBD driver)



On 27/04/13 10:35, Steven Haigh wrote:
> On 27/04/2013 5:51 PM, Steven Haigh wrote:
>> On 27/04/2013 5:06 PM, Roger Pau Monné wrote:
>>> On 27/04/13 03:57, Steven Haigh wrote:
>>>> On 27/04/2013 12:16 AM, Steven Haigh wrote:
>>>>> On 27/04/2013 12:06 AM, Roger Pau Monné wrote:
>>>>>> On 23/04/13 21:05, Steven Haigh wrote:
>>>>>>> Sorry - resending this to Felipe as well - as I started talking to
>>>>>>> him
>>>>>>> directly previously.
>>>>>>>
>>>>>>> Felipe, to bring you up to date, I've copied over the blkback
>>>>>>> files from
>>>>>>> Rogers indirect kernel over the vanilla 3.8.8 kernel files, built and
>>>>>>> tested. Results below:
>>>>>>>
>>>>>
>>>>> Bringing this into context in a nutshell - results showed about 5MB/sec
>>>>> improvement when using buffered disk access - totalling ~57MB/sec write
>>>>> speed vs ~98MB/sec when using the oflag=direct flag to dd.
>>>>>
>>>>> When talking about back porting a few indirect patches to mainline
>>>>> blkback (3.8.8 atm):
>>>>>>> On 24/04/2013 4:13 AM, Roger Pau Monné wrote:
>>>>>>>> I think it requires a non-trivial amount of work, what you could do
>>>>>>>> as a
>>>>>>>> test is directly replace the affected files with the ones in my
>>>>>>>> tree, it
>>>>>>>> is not optimal, but I don't think it's going to cause problems,
>>>>>>>> and you
>>>>>>>> could at least see if indirect descriptors solve your problem.
>>>>>>>
>>>>>>> Ok, I copied across those files, built, packaged and installed
>>>>>>> them on
>>>>>>> my Dom0. Good news is that its a little quicker, bad news is not by
>>>>>>> much.
>>>>>>
>>>>>> Could you try increasing xen_blkif_max_segments variable in
>>>>>> xen-blkfront.c to 64 or 128? It is set to 32 by default. You will only
>>>>>> need to recompile the DomU kernel after this, the Dom0 is able to
>>>>>> support up to 256 indirect segments.
>>>>>
>>>>> I'll have to look at this. All DomU's are Scientific Linux 6.4
>>>>> systems -
>>>>> so essentially RHEL6.4 and so on. I haven't built a RH kernel as yet -
>>>>> so I'll have to look at what is involved. It might be as simple as
>>>>> rebuilding a normal SRPM.
>>>>
>>>> Ok, I've had a look at the RH xen-blkfront.c - and I can't see any
>>>> definition of xen_blkif_max_segments - or anything close. I've attached
>>>> the version used in the EL6 kernel from the kernel-2.6.32-358.6.1.el6
>>>> srpm.
>>>>
>>>> Any ideas on where to go from here?
>>>
>>> I thought you were using the 3.8.x kernel inside the DomU also, if you
>>> are not using it, then it's normal that there's no speed difference, you
>>> have a Dom0 kernel that supports indirect descriptors, but your DomU
>>> doesn't. You must use a kernel that supports indirect descriptors in
>>> both Dom0 and DomU in order to make use of this feature.
>>
>> Ahhh - sorry - I should have been clearer - The Dom0 is kernel 3.8.x
>> (3.8.8 right now) - however the DomUs are all stock EL6 kernels.
>>
>> Hmmmm - I believe the kernel I build for Dom0 *should* work as a DomU.
>> I'll do some more experimentation and see if I can get it working
>> properly as a DomU kernel.
> 
> Ok - now for testing the basics.
> 
> Same tests using vanilla 3.8.8 kernel:
> # dd if=/dev/zero of=output.zero bs=1M count=2048
> 2048+0 records in
> 2048+0 records out
> 2147483648 bytes (2.1 GB) copied, 37.1206 s, 57.9 MB/s
> 
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>             0.00    0.00    2.65    0.00    0.22   97.13
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
> avgrq-sz avgqu-sz   await  svctm  %util
> sdf             395.81  2201.32   27.59  156.95     1.65     9.21 
> 120.60     1.13    6.12   1.19  22.05
> sde             404.86  2208.83   28.04  157.40     1.69     9.24 
> 120.77     1.32    7.15   1.31  24.24
> sdc             435.54  2174.83   30.68  155.63     1.82     9.10 
> 120.09     0.97    5.20   1.11  20.64
> sdd             388.96  2177.26   26.71  155.41     1.62     9.11 
> 120.74     1.10    6.01   1.30  23.60
> md2               0.00     0.00    0.00  537.31     0.00    17.59 
> 67.05     0.00    0.00   0.00   0.00
> 
> # dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct
> 2048+0 records in
> 2048+0 records out
> 2147483648 bytes (2.1 GB) copied, 25.3928 s, 84.6 MB/s
> 
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>             0.22    0.00   15.74    0.00    0.22   83.81
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
> avgrq-sz avgqu-sz   await  svctm  %util
> sdf             336.81 12659.65   10.86  232.59     1.36    50.36 
> 435.06     1.00    4.09   2.51  61.06
> sde            1684.04 11000.22   54.32  189.14     6.79    43.71 
> 424.80     1.45    5.95   3.50  85.28
> sdc             144.35 11177.61    4.66  238.80     0.58    44.60 
> 380.04     0.41    1.70   1.07  26.08
> sdd              20.62 12876.50    0.67  242.79     0.08    51.25 
> 431.80     0.45    1.84   1.15  27.92
> md2               0.00     0.00    0.00 2680.71     0.00    86.47 
> 66.06     0.00    0.00   0.00   0.00
> 
> Installed and rebooted into the patched version I build by just copying 
> the blkback files across to the 3.8.8 tree and building:

Did you also copy xen-blkfront?

> 
> # dd if=/dev/zero of=output.zero bs=1M count=2048
> 2048+0 records in
> 2048+0 records out
> 2147483648 bytes (2.1 GB) copied, 45.2376 s, 47.5 MB/s
> 
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>             0.00    0.00    1.35    0.00    0.45   98.19
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
> avgrq-sz avgqu-sz   await  svctm  %util
> sdd            1340.80  5806.80  158.20  674.40     5.83    25.27 
> 76.51     6.00    7.16   0.80  66.90
> sde            1334.60  5894.00  160.80  686.40     5.86    25.71 
> 76.32     6.87    8.11   0.87  73.52
> sdc            1330.80  5858.20  158.00  682.40     5.86    25.60 
> 76.67     5.71    6.81   0.77  64.84
> sdf            1341.00  5848.80  157.00  681.20     5.83    25.49 
> 76.53     6.23    7.38   0.85  70.92
> md2               0.00     0.00    0.00 1431.40     0.00    46.83 
> 67.01     0.00    0.00   0.00   0.00
> 
> # dd if=/dev/zero of=output.zero bs=1M count=2048 oflag=direct
> 2048+0 records in
> 2048+0 records out
> 2147483648 bytes (2.1 GB) copied, 38.9052 s, 55.2 MB/s
> 
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>             0.00    0.00    5.27    0.00    0.32   94.41
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
> avgrq-sz avgqu-sz   await  svctm  %util
> sdd             493.20  8481.60   36.80  335.80     2.07    34.45 
> 200.73     1.14    3.07   0.97  36.32
> sde            1371.60  7380.20   83.80  304.80     5.66    30.08 
> 188.34     2.20    5.65   1.94  75.38
> sdc             540.20  7556.80   56.00  326.20     2.33    30.80 
> 177.52     1.49    3.90   1.26  48.02
> sdf             734.20  8286.60   64.40  326.20     3.12    33.67 
> 192.92     1.66    4.24   1.45  56.66
> md2               0.00     0.00    0.00 1835.20     0.00    59.20 
> 66.06     0.00    0.00   0.00   0.00
> 
> That is with the same kernel running on both Dom0 and DomU.
> 
> In the dmesg of the DomU, I see the following:
> blkfront: xvdb: flush diskcache: enabled using persistent grants

It seems you are missing some pieces, you should see something like:

blkfront: xvda: flush diskcache: enabled; persistent grants: enabled;
indirect descriptors: enabled;


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.