Thank you for your answer, George.
The origin of my question is more of a business concern than a technical one. Many software products are licensed based on a cost per processor core. It is desirable to sometimes allow customers to pay a fraction of software license costs in exchange for running that software using only a commensurate fraction of available compute power (capacity sub-licensing). If the cap is a means of making a vCPU more-or-less deterministic (in terms of its effective computational capacity) then that would be useful as a programmatic means of enabling capacity sub-licensing. My example below was based on a case where I have a customer that would like to use ‘cap’ to constrain their single vCPU VM to only ½ of a core worth of compute capacity (logically 1/32 of the compute power) in exchange for only paying 1/32 of the license cost for the physical server.
Below you answered:
“You can use ‘cap’ to make the VM in question get 50% of logical vcpu time, which on an idle system will give it 0.5 of the capacity of a physical core (if we don't consider Intel's Turbo Boost technology). But if the system becomes busy, it will get less than 0.5 of the processing capacity of a physical core.”
Are you saying that cap would be able to CONSTRAIN a vCPU to an effective compute capacity equal to 50% of a physical core, but it does not GUARANTEE effective compute capacity equal to 50% of a physical core?
Can you offer any guidance regarding real-world scheduler overhead (when cap>0 is used) and precision (how variable is actual compute power for a vCPU with a cap of 100%, for example)?
- Mike
Michael Palmeter | Sr. Director of Product Management, Oracle Exalogic Elastic Cloud
Phone: +14153736497 | Mobile: +14156949573 | VOIP: +14154027422
Oracle Exalogic Development
200 Oracle Parkway | Redwood Shores, California 94065
| Oracle is committed to developing practices and products that help protect the environment |
Watch the Exalogic 5-minute Demo at http://www.oracle.com/pls/ebn/swf_viewer.load?p_shows_id=12641667
From: George Dunlap [mailto:george.dunlap@xxxxxxxxxxxxx]
Sent: November 15, 2012 10:29 AM
To: Michael Palmeter
Cc: Dario Faggioli; xen-devel@xxxxxxxxxxxxx
Subject: Re: [Xen-devel] Xen credit scheduler question
On 15/11/12 15:43, Michael Palmeter wrote:
Hi all (and Mr. Dunlap in particular),
Haha -- please don't call me "Mr"; I prefer "George", but if you want a title, use "Dr" (since I have PhD). :-)
Example scenario:
- Server hardware: 2 sockets, 8-cores per socket, 2 hardware threads per core (total of 32 hardware threads)
- Test VM: a single virtual machine with a single vCPU, weight=256 and cap=100%
In this scenario, from what I understand, I should be able to load the Test VM with traffic to a maximum of approximately 1/32 of the aggregate compute capacity of the server. The total CPU utilization of the server hardware should be approximately 3.4%, plus the overhead of dom0 (say 1-2). The credits available to any vCPU capped at 100% should be equal to 1/32 of the aggregate compute available for the whole server, correct?
I think to really be precise, you should say, "1/32nd of the logical cpu time available", where "logical cpu time" simply means, "time processing on one logical CPU". At the moment, that is all that either the credit1 or credit2 schedulers look at.
As I'm sure you're aware, not all "logical cpu time" is equal. If one thread of a hyperthread pair is running but the other idle, it will get significantly higher performance than if the other thread is busy. How much is highly unpredictable, and depends very much on exactly what units are shared with the other hyperthread, and the workload running on each unit. But even when both threads are busy, it should (in theory) be rare for both threads to get a throughput of 50%; the whole idea of HT is that threads typically get 70-80% of the full performance of the core (so the overall throughput is increased).
But if course, while this is particularly extreme in the case of hyperthreads, it's also true on a smaller scale even without that -- cores share caches, NUMA nodes share memory bandwidth, and so on. No attempt is made to compensate VMs for cache misses or extra memory latency due to sharing either. :-)
Put simply, is there a way to constrain a VM with 1 vCPU to consume no more than 0.5 of a physical core (hyper-threaded) on the server hardware mentioned below? Does the cap help in that respect?
You can use "cap" to make the VM in question get 50% of logical vcpu time, which on an idle system will give it 0.5 of the capacity of a physical core (if we don't consider Intel's Turbo Boost technology). But if the system becomes busy, it will get less than 0.5 of the processing capacity of a physical core.
I have been struggling to understand how the scheduler can deal with the uncertainty that hyperthreading introduces, however. I know this is an issue that you are tackling in the credit2 scheduler, but I would like to know what your thoughts are on this problem (if you are able to share). Any insight or assistance you could offer would be greatly appreciated.
At the moment it does not attempt to; the only thing it does is try not to schedule two hyperthreads that share a core if there is an idle core. But if there are more active vcpus than cores, then some will share; and the ones that share a core with another vcpu will be charged the same as the ones that have the core all to themselves.
Could you explain why you your question is important to you -- i.e,. what are you trying to accomplish? It sounds a bit like you're more concerned with accuracy in reporting, and control of resources, rather than fairness, for instance.
-George
