Re: [Xen-users] iSCSI initiator on Dom0, exported to DomU via xvd, Disk IO Drops in Half...

["Christopher Chen" <muffaleta@xxxxxxxxx>]

And here's more!

I decided to try something more. I mounted the iSCSI LUN on the Dom0,
created a ext3 file system, and created a image file, 10G big. The
Dom0 has 4G of RAM. The DomU has 2G allocated to it.

So I thought, what the hell...

I block-attached the image file as file://mnt/image xvdc w to the
guest, and mkfs'd it, and mounted, it and ran bonnie to use a 8G test
size (twice the size of the Dom0's RAM, and 4x the DomU's RAM). Here's
the results:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
aenu 8G           130195  22 27364   1           46949   0 313.6   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  6228  99 +++++ +++ +++++ +++  6182  97 +++++ +++ 19075 100
aenu,8G,,,130195,22,27364,1,,,46949,0,313.6,0,16,6228,99,+++++,+++,+++++,+++,6182,97,+++++,+++,19075,100

Yeah, um, so, sustained IO is pretty good via file: driver, but still,
performance via the xvd driver perplexes me:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
aenu 4G           47100   3 15069   0           39072   0  3971   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
aenu,4G,,,47100,3,15069,0,,,39072,0,3971.4,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

So yeah, 36%. Pretty awesome.

I am stumped.

Cheers

cc

On Tue, Jan 13, 2009 at 4:03 PM, Christopher Chen <muffaleta@xxxxxxxxx> wrote:
> On Tue, Jan 13, 2009 at 3:53 PM, Ross Walker <rswwalker@xxxxxxxxx> wrote:
>>
>> On Jan 13, 2009, at 6:37 PM, Ross Walker <rswwalker@xxxxxxxxx> wrote:
>>
>>> On Jan 13, 2009, at 5:48 PM, "Christopher Chen" <muffaleta@xxxxxxxxx>
>>> wrote:
>>>
>>>> Hi there!
>>>>
>>>> I've been wrestling with an issue for a little bit now--
>>>>
>>>> In my test environment, I have tgtd running on a Centos 5.2 box, with
>>>> a raid 10 array backing it.
>>>>
>>>> The initiators are also Centos 5.2 boxes running Xen 3.0.3 userland
>>>> with a Xen 3.1.2/Linux 2.6.18 kernel (as from repos).
>>>>
>>>> Bonnie++ on the Dom0 shows about 110MB/sec writes, and 45MB/sec reads.
>>>
>>> That's kind of lopsided I'd expect it the  other way around.
>>>
>>> Is this hardware RAID on the backend with write-back cache?
>>>
>>>>
>>>> I've attached the iSCSI LUN to the DomU as a virtual block device, and
>>>> I'm seeing 47MB/sec writes, and 39MB/sec reads.
>>>
>>> How did you attach it, what Xen driver did you use phy: or file:?
>>
>> Sorry, missed the virtual block device bit...
>>
>>>> I've tried a few things, like running against a local disk, and
>>>> suprisingly, writes on the DomU are faster than the Dom0--can I assume
>>>> the writes are buffered by the Dom0.
>>>
>>> I'm confused.
>>>
>>> I thought you said above you got 110MB/s on dom0 and 45MB/s on the domU?
>>
>> Never mind my comment, writes are only buffered using file: io, but they are
>> buffered in the domU's page cache which is where you might be seeing the
>> performance difference.
>>
>>>> I'm going to give a shot doing the initialization from the DomU (just
>>>> for kicks...)...and wow! 129MB/sec writes, 49MB/sec reads.
>>>
>>> You've completely lost me now, what do you mean initialization? Do you
>>> mean boot domU off of iSCSI directly?
>>
>> After re-reading I guessed you meant you attached to the iSCSI lun after
>> booting into the VM not as the OS disk.
>>
>> Again you are most likely seeing all cache affect and not the real io.
>>
>>>> This is all with bonnie++ -d /mnt -f -u root:root
>>>>
>>>> Anyone seen this, or have any ideas?
>>>>
>>>> Is any additional latency provided by the xen virtual block device
>>>> causing a degradation in TCP performance (i.e. a window size or
>>>> delayed ACK problem) or is the buffering also causing pain? I'm going
>>>> to keep looking, but I thought I'd ask all of you.
>>>
>>> Any layer you add is going to create latency.
>>>
>>> If you can be a little more clearer I'm sure an accurate explanation can
>>> be made.
>>
>> Try increasing the size of the bonnie test file to defeat the cache, say 2x
>> the memory of the dom0 or domU or target which ever is largest.
>
> The nice thing about bonnie++ -f is it sizes the file for 2x memory.
> These are the numbers. In any case, the ~110MB/sec writes to the iSCSI
> target is our baseline number writing across the network. The Dom0 has
> 4G allocated to it--bonnie++'s test file is 8G. Any reading lower than
> that (in my mind) is degradation. I, of course, expect some effect
> from the layering, but 50%?
>
> cc
>
> --
> Chris Chen <muffaleta@xxxxxxxxx>
> "I want the kind of six pack you can't drink."
> -- Micah
>

-- 
Chris Chen <muffaleta@xxxxxxxxx>
"I want the kind of six pack you can't drink."
-- Micah

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users