Re: [Xen-devel] [for-4.9] Re: HVM guest performance regression

[Juergen Gross <jgross@xxxxxxxx>]

On 06/06/17 18:39, Stefano Stabellini wrote:
> On Tue, 6 Jun 2017, Juergen Gross wrote:
>> On 26/05/17 21:01, Stefano Stabellini wrote:
>>> On Fri, 26 May 2017, Juergen Gross wrote:
>>>> On 26/05/17 18:19, Ian Jackson wrote:
>>>>> Juergen Gross writes ("HVM guest performance regression"):
>>>>>> Looking for the reason of a performance regression of HVM guests under
>>>>>> Xen 4.7 against 4.5 I found the reason to be commit
>>>>>> c26f92b8fce3c9df17f7ef035b54d97cbe931c7a ("libxl: remove freemem_slack")
>>>>>> in Xen 4.6.
>>>>>>
>>>>>> The problem occurred when dom0 had to be ballooned down when starting
>>>>>> the guest. The performance of some micro benchmarks dropped by about
>>>>>> a factor of 2 with above commit.
>>>>>>
>>>>>> Interesting point is that the performance of the guest will depend on
>>>>>> the amount of free memory being available at guest creation time.
>>>>>> When there was barely enough memory available for starting the guest
>>>>>> the performance will remain low even if memory is being freed later.
>>>>>>
>>>>>> I'd like to suggest we either revert the commit or have some other
>>>>>> mechanism to try to have some reserve free memory when starting a
>>>>>> domain.
>>>>>
>>>>> Oh, dear.  The memory accounting swamp again.  Clearly we are not
>>>>> going to drain that swamp now, but I don't like regressions.
>>>>>
>>>>> I am not opposed to reverting that commit.  I was a bit iffy about it
>>>>> at the time; and according to the removal commit message, it was
>>>>> basically removed because it was a piece of cargo cult for which we
>>>>> had no justification in any of our records.
>>>>>
>>>>> Indeed I think fixing this is a candidate for 4.9.
>>>>>
>>>>> Do you know the mechanism by which the freemem slack helps ?  I think
>>>>> that would be a prerequisite for reverting this.  That way we can have
>>>>> an understanding of why we are doing things, rather than just
>>>>> flailing at random...
>>>>
>>>> I wish I would understand it.
>>>>
>>>> One candidate would be 2M/1G pages being possible with enough free
>>>> memory, but I haven't proofed this yet. I can have a try by disabling
>>>> big pages in the hypervisor.
>>>
>>> Right, if I had to bet, I would put my money on superpages shattering
>>> being the cause of the problem.
>>
>> Seems you would have lost your money...
>>
>> Meanwhile I've found a way to get the "good" performance in the micro
>> benchmark. Unfortunately this requires to switch off the pv interfaces
>> in the HVM guest via "xen_nopv" kernel boot parameter.
>>
>> I have verified that pv spinlocks are not to blame (via "xen_nopvspin"
>> kernel boot parameter). Switching to clocksource TSC in the running
>> system doesn't help either.
> 
> What about xen_hvm_exit_mmap (an optimization for shadow pagetables) and
> xen_hvm_smp_init (PV IPI)?

xen_hvm_exit_mmap isn't active (kernel message telling me so was
issued).

>> Unfortunately the kernel seems no longer to be functional when I try to
>> tweak it not to use the PVHVM enhancements.
> 
> I guess you are not talking about regular PV drivers like netfront and
> blkfront, right?

The plan was to be able to use PV drivers without having to use PV
callbacks and PV timers. This isn't possible right now.

>> I'm wondering now whether
>> there have ever been any benchmarks to proof PVHVM really being faster
>> than non-PVHVM? My findings seem to suggest there might be a huge
>> performance gap with PVHVM. OTOH this might depend on hardware and other
>> factors.
>>
>> Stefano, didn't you do the PVHVM stuff back in 2010? Do you have any
>> data from then regarding performance figures?
> 
> Yes, I still have these slides:
> 
> https://www.slideshare.net/xen_com_mgr/linux-pv-on-hvm

Thanks. So you measured the overall package, not the single items like
callbacks, timers, time source? I'm asking because I start to believe
there are some of those slower than their non-PV variants.


Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel