Thanks very much George. Very helpful indeed.
- Mike
Michael Palmeter | Sr. Director of Product Management, Oracle Exalogic Elastic Cloud
Phone: +14153736497 | Mobile: +14156949573 | VOIP: +14154027422
Oracle Exalogic Development
200 Oracle Parkway | Redwood Shores, California 94065
| Oracle is committed to developing practices and products that help protect the environment |
Watch the Exalogic 5-minute Demo at http://www.oracle.com/pls/ebn/swf_viewer.load?p_shows_id=12641667
From: George Dunlap [mailto:george.dunlap@xxxxxxxxxxxxx]
Sent: November 15, 2012 11:53 AM
To: Michael Palmeter
Cc: Ashok Aletty; Dario Faggioli; xen-devel@xxxxxxxxxxxxx
Subject: Re: [Xen-devel] Xen credit scheduler question
On 15/11/12 19:03, Michael Palmeter wrote:
Thank you for your answer, George.
The origin of my question is more of a business concern than a technical one. Many software products are licensed based on a cost per processor core. It is desirable to sometimes allow customers to pay a fraction of software license costs in exchange for running that software using only a commensurate fraction of available compute power (capacity sub-licensing). If the cap is a means of making a vCPU more-or-less deterministic (in terms of its effective computational capacity) then that would be useful as a programmatic means of enabling capacity sub-licensing. My example below was based on a case where I have a customer that would like to use ‘cap’ to constrain their single vCPU VM to only ½ of a core worth of compute capacity (logically 1/32 of the compute power) in exchange for only paying 1/32 of the license cost for the physical server.
Right -- I've seen the "limit cpu power for licensing purposes" thing before, but I think that only went down to cores, not sub-core.
Below you answered:
“You can use ‘cap’ to make the VM in question get 50% of logical vcpu time, which on an idle system will give it 0.5 of the capacity of a physical core (if we don't consider Intel's Turbo Boost technology). But if the system becomes busy, it will get less than 0.5 of the processing capacity of a physical core.”
Are you saying that cap would be able to CONSTRAIN a vCPU to an effective compute capacity equal to 50% of a physical core, but it does not GUARANTEE effective compute capacity equal to 50% of a physical core?
Theoretically, a cap at 50 will give your single-vcpu VM 50% of the time of one hyperthread.
So if C is "typicall throughput of a single non-hyperthreaded core running at standard requency", and we factor out Turbo Boost, then there are two cases to consider:
* Other thread is idle. In that case, the VM will get 0.5C.
* The other thread is busy. In this case, assuming a 0.7 factor, the VM will get 0.5 * (0.7 * C), or about 0.35C
So the total computing power available to the VM should be <= 0.5C (satisfying the licensing requirements), but on a busy system it may be significantly less than 0.5C (perhaps not so satisfying to the owner of the VM).
I don't think it should be terribly difficult to put a simple "shared hyperthread" multiplier on the credit burned -- if someone at Oracle wanted to help implement this, we'd be happy to point you in the right direction. :-)
If you have Turbo Boost, then (as I understand it) the CPU can raise the clock speed of the processor when threads or cores are idle; the wikipedia article seems to think some processors can increase the clock speed up to 1.6x over the baseline frequency. That would throw a bit of a wrench in the works, as you might end up with 0.5 * 1.6 * C = 0.8 C > 0.5 C; however, looking at Intel's website, it looks like only 2- and 4-core processors have TurboBoost, so maybe on 8-core processors we can punt on that thorny issue for a little while yet. :-)
Can you offer any guidance regarding real-world scheduler overhead (when cap>0 is used) and precision (how variable is actual compute power for a vCPU with a cap of 100%, for example)?
I have not done extensive testing with the cap; I mainly know the mechanism by which it works. There is no extra accounting done in the scheduler for having a cap: all vcpus are assigned credit every 30ms according to their weight and cap. The difference is that if a non-capped vcpu uses up its credits, it is allowed to go negative; whereas a capped vcpu will be paused until it receives more credits. So there should be no extra hypervisor overhead from using a cap.
The cap fundamentally works by locking out a vcpu for very small amounts of time within the 30ms accounting window. But this same effect might happen just by having other VMs competing for the cpu; so in theory shouldn't be any riskier than virtualizing in the first place.
Executive summary: Factoring out Turbo Boost, "cap" should be able to set a sub-core upper-bound on processing power. But on a busy system, it may result in the VM getting less than its upper-bound in processing power.
However, scheduling is a very complex and dynamic system, and like economics, very simple changes can have unpredictable results. So it's probably a good idea to do some testing before recommending it to customers. :-)
BTW, are you familiar with Xen's cpupool functionality? The guys at Fujitsu wrote it so that a customer could rent a fixed number of cores to a customer, who could then run as many VMs on those cores as they wanted. I think licensing restrictions had something to do with that as well. More about that here, if you're interested:
http://blog.xen.org/index.php/2012/04/23/xen-4-2-cpupools/
-George
