Re: [Xen-users] Xen IO performance issues

[Juergen Gross <jgross@xxxxxxxx>]

On 20/09/2018 11:49, marki wrote:
> Hello,
> 
> On 2018-09-19 21:43, Hans van Kranenburg wrote:
>> On 09/19/2018 09:19 PM, marki wrote:
>>> On 2018-09-19 20:35, Sarah Newman wrote:
>>>> On 09/14/2018 04:04 AM, marki wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> We're having trouble with a dd "benchmark". Even though that probably
>>>>> doesn't mean much since multiple concurrent jobs using a benckmark
>>>>> like FIO for
>>>>> example work ok, I'd like to understand where the bottleneck is / why
>>>>> this behaves differently.
>>>>>
>>>>> Now in a Xen DomU running kernel 4.4 it looks like the following and
>>>>> speed is low / not what we're used to:
>>>>>
>>>>> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
>>>>> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
>>>>> dm-0              0.00     0.00    0.00  100.00     0.00    99.00 
>>>>> 2027.52     1.45   14.56    0.00   14.56  10.00 100.00
>>>>> xvdb              0.00     0.00    0.00 2388.00     0.00    99.44   
>>>>> 85.28    11.74    4.92    0.00    4.92   0.42  99.20
>>>>>
>>>>> # dd if=/dev/zero of=/u01/dd-test-file bs=32k count=250000
>>>>> 1376059392 bytes (1.4 GB, 1.3 GiB) copied, 7.09965 s, 194 MB/s
>>
>> Interesting.
>>
>> * Which Xen version are you using?
> 
> That particular version was XenServer 7.1 LTSR (Citrix). We also tried
> the newer current release 7.6, makes no difference.
> Before you start screaming:
> XS eval licenses do not contain any support so we can't ask them.
> People in Citrix discussion forums are nice but don't seem to know
> details necessary to solve this.
> 
>> * Which Linux kernel version is being used in the dom0?
> 
> In 7.1 it is "4.4.0+2".
> In 7.6 that would be "4.4.0+10".
> 
>> * Is this a PV, HVM or PVH guest?
> 
> In any case blkfront (and thus blkback) were being used (which seems to
> transfer data by that ring structure I mentioned and which explains the
> small block size albeit not necessarily the low queue depth).
> 
>> * ...more details you can share?
> 
> Well, not much more except that we are talking about Suse Enterprise
> Linux 12 up to SP3 in the DomU here. We also tried RHEL 7.5 and the
> result (slow single-threaded writes) was the same. Reads are not
> blazingly fast either BTW.
> 
>>
>>>>> Note the low queue depth on the LVM device and additionally the low
>>>>> request size on the virtual disk.
>>>>>
>>>>> (As in the ESXi VM there's an LVM layer inside the DomU but it
>>>>> doesn't matter whether it's there or not.)
>>>>>
>>>>>
>>>>> The above applies to HV + HVPVM modes using kernel 4.4 in the DomU.
>>
>> Do you mean PV and PVHVM, instead?
>>
> 
> Oups yes, in any case blkfront (and thus blkback) were being used.
> 
>>
>> What happens when you use a recent linux kernel in the guest, like 4.18?
> 
> I'd have to get back to you on that. However, as long as blkback stays
> the same I'm not sure what would happen.
> In any case we'd want to stick with the OSes that the XS people support,
> I'll have to find out if there are some with more recent kernels than
> SLES or RHEL.

I have just done a small test for other purposes requiring to do reads
in a domU using blkfront/blkback. The data was cached in dom0, so the
only limiting factor was cpu/memory speed and the block ring interface
of Xen. I was able to transfer 1.8 GB/s on a laptop with a dual core
i7-4600M CPU @ 2.90GHz.

So I don't think the ring buffer interface is a real issue here.

Kernels (in domU and dom0) are 4.19-rc5, Xen is 4.12-unstable.

Using a standard SLE12-SP2 domU (kernel 4.4.121) with the same dom0
as in the test before returned the same result.


Juergen

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-users