[Xen-devel] [RFC] Add static priority into credit scheduler

["Su, Disheng" <disheng.su@xxxxxxxxx>]

Hi all,
        Attached patches add static priority into credit scheduler.
        Currently, credit scheduler has 4 kinds of priority: BOOST, UNDER, OVER 
and IDLE. And the priority of VM is dynamically changed according to the credit 
of VM, or I/O events, the highest priority VM is chosed to be scheduled in for 
each scheduling period. Due to priority is not fixed, which VM will be 
scheduled in is properly unknown. The I/O latency caused by scheduler is well 
analyzed in [1] and [2]. They provides ways to reduce I/O latency and also 
retain CPU and I/O fairness between VMs to some extend.
        There are some cases that reducing latency is much preferable to CPU or 
I/O fairness, such as RTOS guest or VM with device(audio)-assigned. The 
straightforward way is to set static(fixed) highest priority for this VM, to 
make sure it is scheduled each time. Attached patches implemented this kind of 
mechanism, like SCHED_RR/SCHED_FIFO in Linux.
        
        How it works?
        --Users can set RT priority(between 1~100) for domains. The larger the 
number, the higher the priority. Users can also change a RT domain into a 
non-RT domain by setting its priority other than 1~100.  
        --Scheduler always chooses the highest priority domain to run for RT 
domains, no changes for non-RT domains in there. If RT domains have the same 
priority, round robin between this domains for every 30ms. 30ms is the default 
scheduling period, it can be changed to 2ms or other value if needed. 
        --There is still accounting for current running non-RT vcpu in every 
10ms, accounting for all non-RT domains in every 30ms as credit scheduler did 
before. 

        Implementation details:
         -- In order to minimize the modification in the credit scheduler, one 
additional rt runqueue per pcpu is added, and one rt active domain list added 
in csched_private. RT vcpus are added into the rt runqueue in the running pcpu, 
and rt domains are added into rt active domain.
         -- Scheduler always chooses the highest priority in the rt runqueue if 
it's not empty at first, then chooses from normal runqueue instead.
         --__runq_insert/__runq_remove are changed to based on the priority of 
vcpu.
         -- Vcpu accounting is only took effects on the non-RT vcpus as before. 
Non-RT vcpus propotionally share the rest of cpu based on their weight. The 
total weight is changed during adding/removing RT domains, e.g. promoting a 
non-RT domain to a RT domain, total weight is  substracted by the weight of 
non-RT domain.
        
        How to use it:
                set priority(y) of a VM(x) by: "xm sched-credit -d x -p y"
        
        Test results:
        I did some tests with this patches according to following configuration:
                CPU: Intel Core 2 Duo E6850, Xen(1881), 7 VMs created on one 
physical machine A, each 2 VMs pair ping with each other, the other VM has RT 
priority. Another physical machine B connects with it through 1G network card 
directly. Conduct these tests from B to A, e.g ping A from B.
        some test results are uploaded to 
http://wiki.xensource.com/xenwiki/DishengSu, FYI.

        Summary:
        This patches minimize the scheduling latency, while losing CPU, or I/O 
fairness. It can be used as a scheduler for RT guest, for some cases(such as RT 
guest and non-RT guests co-exist). While there are lot of areas to improve real 
time response, such as interrupt latency, Xen I/O model[3].
        Any comments are appreciated. Thanks!

---------------------
[1]Scheduling I/O in Virtual Machine Monitors
[2]Evaluation and Consideration of the Credit Scheduler for Client 
Virtualization 
[3]A step to support real-time in virtual machine

Best Regards,
Disheng, Su

Attachment: static_priority_for_xen.patch
Description: static_priority_for_xen.patch

Attachment: static_priority_for_xen_tools.patch
Description: static_priority_for_xen_tools.patch

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel