[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [PATCH 0/2] xen: credit2: fix vcpu starvation due to too few credits

On 12.03.20 14:44, Dario Faggioli wrote:
Hello everyone,

There have been reports of a Credit2 issue due to which vCPUs where
being starved, to the point that guest kernel would complain or even

See the following xen-users and xen-devel threads:

I did some investigations, and figured out that the vCPUs in question
are not scheduled for long time intervals because they somehow manage to
be given an amount of credits which is less than the credit the idle
vCPU has.

An example of this situation is shown here. In fact, we can see d0v1
sitting in the runqueue while all the CPUs are idle, as it has
-1254238270 credits, which is smaller than -2^30 = −1073741824:

     (XEN) Runqueue 0:
     (XEN)   ncpus              = 28
     (XEN)   cpus               = 0-27
     (XEN)   max_weight         = 256
     (XEN)   pick_bias          = 22
     (XEN)   instload           = 1
     (XEN)   aveload            = 293391 (~111%)
     (XEN)   idlers: 00,00000000,00000000,00000000,00000000,00000000,0fffffff
     (XEN)   tickled: 00,00000000,00000000,00000000,00000000,00000000,00000000
     (XEN)   fully idle cores: 
     (XEN) Runqueue 0:
     (XEN) CPU[00] runq=0, sibling=00,..., core=00,...
     (XEN) CPU[01] runq=0, sibling=00,..., core=00,...
     (XEN) CPU[26] runq=0, sibling=00,..., core=00,...
     (XEN) CPU[27] runq=0, sibling=00,..., core=00,...
     (XEN) RUNQ:
     (XEN)     0: [0.1] flags=0 cpu=5 credit=-1254238270 [w=256] load=262144 

This happens bacause --although very rarely-- vCPUs are allowed to
execute for much more than the scheduler would want them to.

For example, I have a trace showing that csched2_schedule() is invoked at
t=57970746155ns. At t=57970747658ns (+1503ns) the s_timer is set to
fire at t=57979485083ns, i.e., 8738928ns in future. That's because credit
of snext is exactly that 8738928ns. Then, what I see is that the next
call to burn_credits(), coming from csched2_schedule() for the same vCPU
happens at t=60083283617ns. That is *a lot* (2103798534ns) later than
when we expected and asked. Of course, that also means that delta is
2112537462ns, and therefore credits will sink to -2103798534!

Current ideas are:

- Could it be the vcpu is busy for very long time in the hypervisor?
  So either fighting with another vcpu for a lock, doing a long
  running hypercall, ...

- The timer used is not reliable.

- The time base is not reliable (tsc or whatever is used for getting
  the time has jumped 2 seconds into the future).

- System management mode has kicked in.


Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.