Re: [Xen-devel] Xen credit scheduler question

[George Dunlap <george.dunlap@xxxxxxxxxxxxx>]

On 15/11/12 19:03, Michael Palmeter wrote:

Thank you for your answer, George.

 

The origin of my question is more of a business concern than a technical one.  Many software products are licensed based on a cost per processor core.  It is desirable to sometimes allow customers to pay a fraction of software license costs in exchange for running that software using only a commensurate fraction of available compute power (capacity sub-licensing).  If the cap is a means of making a vCPU more-or-less deterministic (in terms of its effective computational capacity) then that would be useful as a programmatic means of enabling capacity sub-licensing.  My example below was based on a case where I have a customer that would like to use ‘cap’ to constrain their single vCPU VM to only ½ of a core worth of compute capacity (logically 1/32 of the compute power) in exchange for only paying 1/32 of the license cost for the physical server.

Right -- I've seen the "limit cpu power for licensing purposes" thing before, but I think that only went down to cores, not sub-core.

 

Below you answered:

 

“You can use ‘cap’ to make the VM in question get 50% of logical vcpu time, which on an idle system will give it 0.5 of the capacity of a physical core (if we don't consider Intel's Turbo Boost technology).  But if the system becomes busy, it will get less than 0.5 of the processing capacity of a physical core.”

 

Are you saying that cap would be able to CONSTRAIN a vCPU to an effective compute capacity equal to 50% of a physical core, but it does not GUARANTEE effective compute capacity equal to 50% of a physical core?

Theoretically, a cap at 50 will give your single-vcpu VM 50% of the time of one hyperthread.

So if C is "typicall throughput of a single non-hyperthreaded core running at standard requency", and we factor out Turbo Boost, then there are two cases to consider:

* Other thread is idle.  In that case, the VM will get 0.5C.
* The other thread is busy.  In this case, assuming a 0.7 factor, the VM will get 0.5 * (0.7 * C), or about 0.35C

So the total computing power available to the VM should be <= 0.5C (satisfying the licensing requirements), but on a busy system it may be significantly less than 0.5C (perhaps not so satisfying to the owner of the VM).

I don't think it should be terribly difficult to put a simple "shared hyperthread" multiplier on the credit burned -- if someone at Oracle wanted to help implement this, we'd be happy to point you in the right direction. :-)

If you have Turbo Boost, then (as I understand it) the CPU can raise the clock speed of the processor when threads or cores are idle; the wikipedia article seems to think some processors can increase the clock speed up to 1.6x over the baseline frequency.  That would throw a bit of a wrench in the works, as you might end up with 0.5 * 1.6 * C = 0.8 C > 0.5 C; however, looking at Intel's website, it looks like only 2- and 4-core processors have TurboBoost, so maybe on 8-core processors we can punt on that thorny issue for a little while yet. :-)

Can you offer any guidance regarding real-world scheduler overhead (when cap>0 is used) and precision (how variable is actual compute power for a vCPU with a cap of 100%, for example)?

I have not done extensive testing with the cap; I mainly know the mechanism by which it works.  There is no extra accounting done in the scheduler for having a cap: all vcpus are assigned credit every 30ms according to their weight and cap.  The difference is that if a non-capped vcpu uses up its credits, it is allowed to go negative; whereas a capped vcpu will be paused until it receives more credits.  So there should be no extra hypervisor overhead from using a cap.

The cap fundamentally works by locking out a vcpu for very small amounts of time within the 30ms accounting window.  But this same effect might happen just by having other VMs competing for the cpu; so in theory shouldn't be any riskier than virtualizing in the first place.

Executive summary: Factoring out Turbo Boost, "cap" should be able to set a sub-core upper-bound on processing power.  But on a busy system, it may result in the VM getting less than its upper-bound in processing power.

However, scheduling is a very complex and dynamic system, and like economics, very simple changes can have unpredictable results.  So it's probably a good idea to do some testing before recommending it to customers. :-)

BTW, are you familiar with Xen's cpupool functionality?  The guys at Fujitsu wrote it so that a customer could rent a fixed number of cores to a customer, who could then run as many VMs on those cores as they wanted.  I think licensing restrictions had something to do with that as well.  More about that here, if you're interested:
 http://blog.xen.org/index.php/2012/04/23/xen-4-2-cpupools/

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel