[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Xen credit scheduler question

Thanks very much George.  Very helpful indeed.


- Mike

Michael Palmeter | Sr. Director of Product Management, Oracle Exalogic Elastic Cloud
Phone: +14153736497 | Mobile: +14156949573 | VOIP: +14154027422
Oracle Exalogic Development
200 Oracle Parkway | Redwood Shores, California 94065

Green Oracle

Oracle is committed to developing practices and products that help protect the environment


Watch the Exalogic 5-minute Demo at http://www.oracle.com/pls/ebn/swf_viewer.load?p_shows_id=12641667


From: George Dunlap [mailto:george.dunlap@xxxxxxxxxxxxx]
Sent: November 15, 2012 11:53 AM
To: Michael Palmeter
Cc: Ashok Aletty; Dario Faggioli; xen-devel@xxxxxxxxxxxxx
Subject: Re: [Xen-devel] Xen credit scheduler question


On 15/11/12 19:03, Michael Palmeter wrote:

Thank you for your answer, George.


The origin of my question is more of a business concern than a technical one.  Many software products are licensed based on a cost per processor core.  It is desirable to sometimes allow customers to pay a fraction of software license costs in exchange for running that software using only a commensurate fraction of available compute power (capacity sub-licensing).  If the cap is a means of making a vCPU more-or-less deterministic (in terms of its effective computational capacity) then that would be useful as a programmatic means of enabling capacity sub-licensing.  My example below was based on a case where I have a customer that would like to use ‘cap’ to constrain their single vCPU VM to only ½ of a core worth of compute capacity (logically 1/32 of the compute power) in exchange for only paying 1/32 of the license cost for the physical server.

Right -- I've seen the "limit cpu power for licensing purposes" thing before, but I think that only went down to cores, not sub-core.


Below you answered:


“You can use ‘cap’ to make the VM in question get 50% of logical vcpu time, which on an idle system will give it 0.5 of the capacity of a physical core (if we don't consider Intel's Turbo Boost technology).  But if the system becomes busy, it will get less than 0.5 of the processing capacity of a physical core.”


Are you saying that cap would be able to CONSTRAIN a vCPU to an effective compute capacity equal to 50% of a physical core, but it does not GUARANTEE effective compute capacity equal to 50% of a physical core?

Theoretically, a cap at 50 will give your single-vcpu VM 50% of the time of one hyperthread.

So if C is "typicall throughput of a single non-hyperthreaded core running at standard requency", and we factor out Turbo Boost, then there are two cases to consider:

* Other thread is idle.  In that case, the VM will get 0.5C.
* The other thread is busy.  In this case, assuming a 0.7 factor, the VM will get 0.5 * (0.7 * C), or about 0.35C

So the total computing power available to the VM should be <= 0.5C (satisfying the licensing requirements), but on a busy system it may be significantly less than 0.5C (perhaps not so satisfying to the owner of the VM).

I don't think it should be terribly difficult to put a simple "shared hyperthread" multiplier on the credit burned -- if someone at Oracle wanted to help implement this, we'd be happy to point you in the right direction. :-)

If you have Turbo Boost, then (as I understand it) the CPU can raise the clock speed of the processor when threads or cores are idle; the wikipedia article seems to think some processors can increase the clock speed up to 1.6x over the baseline frequency.  That would throw a bit of a wrench in the works, as you might end up with 0.5 * 1.6 * C = 0.8 C > 0.5 C; however, looking at Intel's website, it looks like only 2- and 4-core processors have TurboBoost, so maybe on 8-core processors we can punt on that thorny issue for a little while yet. :-)

Can you offer any guidance regarding real-world scheduler overhead (when cap>0 is used) and precision (how variable is actual compute power for a vCPU with a cap of 100%, for example)?

I have not done extensive testing with the cap; I mainly know the mechanism by which it works.  There is no extra accounting done in the scheduler for having a cap: all vcpus are assigned credit every 30ms according to their weight and cap.  The difference is that if a non-capped vcpu uses up its credits, it is allowed to go negative; whereas a capped vcpu will be paused until it receives more credits.  So there should be no extra hypervisor overhead from using a cap.

The cap fundamentally works by locking out a vcpu for very small amounts of time within the 30ms accounting window.  But this same effect might happen just by having other VMs competing for the cpu; so in theory shouldn't be any riskier than virtualizing in the first place.

Executive summary: Factoring out Turbo Boost, "cap" should be able to set a sub-core upper-bound on processing power.  But on a busy system, it may result in the VM getting less than its upper-bound in processing power.

However, scheduling is a very complex and dynamic system, and like economics, very simple changes can have unpredictable results.  So it's probably a good idea to do some testing before recommending it to customers. :-)

BTW, are you familiar with Xen's cpupool functionality?  The guys at Fujitsu wrote it so that a customer could rent a fixed number of cores to a customer, who could then run as many VMs on those cores as they wanted.  I think licensing restrictions had something to do with that as well.  More about that here, if you're interested:


Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.