[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Xen credit scheduler question

On 15/11/12 15:43, Michael Palmeter wrote:

Hi all (and Mr. Dunlap in particular),

Haha -- please don't call me "Mr"; I prefer "George", but if you want a title, use "Dr" (since I have  PhD). :-)

Example scenario:


  • Server hardware: 2 sockets, 8-cores per socket, 2 hardware threads per core (total of 32 hardware threads)
  • Test VM: a single virtual machine with a single vCPU, weight=256 and cap=100%


In this scenario, from what I understand, I should be able to load the Test VM with traffic to a maximum of approximately 1/32 of the aggregate compute capacity of the server.  The total CPU utilization of the server hardware should be approximately 3.4%, plus the overhead of dom0 (say 1-2).  The credits available to any vCPU capped at 100% should be equal to 1/32 of the aggregate compute available for the whole server, correct?

I think to really be precise, you should say, "1/32nd of the logical cpu time available", where "logical cpu time" simply means, "time processing on one logical CPU".  At the moment, that is all that either the credit1 or credit2 schedulers look at.

As I'm sure you're aware, not all "logical cpu time" is equal.  If one thread of a hyperthread pair is running but the other idle, it will get significantly higher performance than if the other thread is busy.  How much is highly unpredictable, and depends very much on exactly what units are shared with the other hyperthread, and the workload running on each unit.  But even when both threads are busy, it should (in theory) be rare for both threads to get a throughput of 50%; the whole idea of HT is that threads typically get 70-80% of the full performance of the core (so the overall throughput is increased).

But if course, while this is particularly extreme in the case of hyperthreads, it's also true on a smaller scale even without that -- cores share caches, NUMA nodes share memory bandwidth, and so on.  No attempt is made to compensate VMs for cache misses or extra memory latency due to sharing either. :-)

Put simply, is there a way to constrain a VM with 1 vCPU to consume no more than 0.5 of a physical core (hyper-threaded) on the server hardware mentioned below? Does the cap help in that respect?

You can use "cap" to make the VM in question get 50% of logical vcpu time, which on an idle system will give it 0.5 of the capacity of a physical core (if we don't consider Intel's Turbo Boost technology).  But if the system becomes busy, it will get less than 0.5 of the processing capacity of a physical core.

I have been struggling to understand how the scheduler can deal with the uncertainty that hyperthreading introduces, however.  I know this is an issue that you are tackling in the credit2 scheduler, but I would like to know what your thoughts are on this problem (if you are able to share).  Any insight or assistance you could offer would be greatly appreciated. 

At the moment it does not attempt to; the only thing it does is try not to schedule two hyperthreads that share a core if there is an idle core.  But if there are more active vcpus than cores, then some will share; and the ones that share a core with another vcpu will be charged the same as the ones that have the core all to themselves.

Could you explain why you your question is important to you -- i.e,. what are you trying to accomplish?  It sounds a bit like you're more concerned with accuracy in reporting, and control of resources, rather than fairness, for instance.

Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.