[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] apic-v reduce network performance in my test case

On 2015/2/6 3:26, Konrad Rzeszutek Wilk wrote:
On Thu, Feb 05, 2015 at 11:04:37AM +0800, Liuqiming (John) wrote:
> On 2015/2/5 10:57, Liuqiming (John) wrote:
>> sorry for late replay
>> On 2015/2/3 23:32, Konrad Rzeszutek Wilk wrote:
>>> On Tue, Feb 03, 2015 at 10:24:08AM +0800, Liuqiming (John) wrote:
>>>> On 2015/2/2 22:59, Konrad Rzeszutek Wilk wrote:
>>>>> On Mon, Feb 02, 2015 at 09:58:57PM +0800, Liuqiming (John) wrote:
>>>>>> Hi Jan,
>>>>>> Thanks for the reply.
>>>>>> On 2015/2/2 18:12, Jan Beulich wrote:
>>>>>>>>>> On 31.01.15 at 11:29, <john.liuqiming@xxxxxxxxxx> wrote:
>>>>>>>>          Recently I met an odd performance problem: when I turn
>>> on APIC
>>>>>>>> Virtualization feature (apicv=1), the network performance of a
>>> windows
>>>>>>>> guest become worse.
>>>>>>>>          My test case like this: host only have one windows 2008
>>> R2 HVM
>>>>>>>> guest running,and this guest has a SR-IOV VF network passthrough
>>> to it.
>>>>>>>> Guest using this network access a NAS device. No fontend or
>>> backend of
>>>>>>>> network and storage, all data transfered through network.
>>>>>>>>          The xentrace data shows: the mainly difference between
>>> apicv and
>>>>>>>> non-apicv, is the way guest write apic registers, and
>>>>>>>> EXIT_REASON_MSR_WRITE vmexit cost much more time than
>>>>>>>> EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit
>>> is much
>>>>>>>> less than using APIC-v.
>>>>>>> There being heavier use of the pause VMEXIT doesn't by itself say
>>>>>>> anything, I'm afraid. It may suggest that you have a C-state exit
>>>>>>> latency problem - try lowering the maximum C-state allowed, or
>>>>>>> disabling use of C-states altogether.
>>>>>> Sorry, I forgot to mention  my test scenario:
>>>>>> Its a video test suite,I am not sure what the logic inside the
>>> tools exactly (not opensource tool).
>>>>>> The basic flow is:
>>>>>>         1) test suite start several thread to read video file from
>>> disk (from NAS through network in my case)
>>>>>>         2) decode these video data as a frame one by one
>>>>>>         3) if  any frame delay more than 40ms, then mark as lost
>>>>>> test result:
>>>>>>           apicv=1,  there can be 15 thread running at the same time
>>> without lost frame
>>>>>>           apicv=0,  there can be 22 thread running at the same time
>>> without lost frame
>>>>>> so when I'm saying apicv reduce the performance, I got the
>>> conclusion from the test result not from what xentrace shows.
>>>>>>>> In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
>>>>>>>> "VMX/Viridian: suppress MSR-based APIC suggestion when having
>>> APIC-V",
>>>>>>>> msr based apic is disabled when apic-v is on, I wonder can they
>>> co-exist
>>>>>>>> in some way? seems for windows guest msr-based apic has better
>>> performance.
>>>>>>> The whole purpose is to avoid the costly MSR access exits. Why
>>>>>>> would you want to reintroduce that overhead?
>>>>>>> Jan
>>>>>> I agree to avoid the MSR access vmexit by using apicv, I just do
>>> not know what's the side effect.
>>>>>> Because from the test result,  apicv replacing  msr-based access
>>> brings performance reduction.
>>>>> It sounds like it brings latency disruption, not neccessarily
>>> performance reduction.
>>>>> What happens if you run with the cpufreq turned to performance, or
>>> as Jan suggested -
>>>>> with disabling C-states?
>>>> I have tried disable C-states in BIOS, the test result is still the
>>> same.
>>>> There is no frontend-backend device in my test, dom0 is not
>>> involved, so you mean cpufreq in DomU, right?
>>> I meant on Xen (cpufreq=performance). But if you disable C-states then
>>> that should
>>> not matter.
>>>> I have no idea how to change cpufreq of a windows os but I will
>>> figure out and try.
>>> That should not be needed.
>>> But back to your problem. When you say you have 'frame delay' - is it
>>> because
>>> of the NAS storage not getting the I/Os fast enough?
>> The  NAS storage should not be the bottleneck, it's a ramdisk connected
>> with 10G network card.
>> Besides, I'am using the same test environment, the only difference is
>> whether apicv=1 or not.
>>> What are you using as network device? Is it an SR-IOV device or an
>>> emulated
>>> device (e1000, ne2k, rtl81..?)?
>> The vm using Intel Corporation 82599 Ethernet Controller Virtual
>> Function passthrough network and has the corresponding driver installed.
>>>>>> _______________________________________________
>>>>>> Xen-devel mailing list
>>>>>> Xen-devel@xxxxxxxxxxxxx
>>>>>> http://lists.xen.org/xen-devel
>>>>> .
>> When  tracing the performance problem, I found whether apicv is on
>> change the result.
>> And after investigating the code, turns out CPUID4A_MSR_BASED_APIC
>> presents or not is the key factor.

Ah, willing to send out a patch for that?
I have not get the conclusion: why CPUID4A_MSR_BASED_APIC affect performance.
But I will continue to dig in, and happy to send out what I found.
> sorry, forget cc the list

for testing purpose, reverting this commit will do:

"VMX/Viridian: suppress MSR-based APIC suggestion when having APIC-V"


Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.