RE: [Xen-devel] unnecessary VCPU migration happens again

["Petersson, Mats" <Mats.Petersson@xxxxxxx>]

> -----Original Message-----
> From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx 
> [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of 
> Emmanuel Ackaouy
> Sent: 06 December 2006 14:02
> To: Xu, Anthony
> Cc: xen-devel@xxxxxxxxxxxxxxxxxxx; xen-ia64-devel
> Subject: Re: [Xen-devel] unnecessary VCPU migration happens again
> 
> Hi Anthony.
> 
> Could you send xentrace output for scheduling operations
> in your setup?
> 
> Perhaps we're being a little too aggressive spreading
> work across sockets. We do this on vcpu_wake right now.
> 
> I'm not sure I understand why HVM VCPUs would block
> and wake more often than PV VCPUs though. Can you
> explain?

Whilst I don't know any of the facts of the original poster, I can tell
you why HVM and PV guests have differing number of scheduling
operations... 

Every time you get a IOIO/MMIO vmexit that leads to a qemu-dm
interaction, you'll get a context switch. So for an average IDE block
read/write (for example) on x86, you get 4-5 IOIO intercepts to send the
command to qemu, then an interrupt is sent to the guest to indicate that
the operation is finished, followed by a 256 x 16-bit IO read/write of
the sector content (which is normally just one IOIO intercept unless the
driver is "stupid"). This means around a dozen or so schedule operations
to do one disk IO operation.

The same operation in PV (or using PV driver in HVM guest of course)
would require a single transaction from DomU to Dom0 and back, so only
two schedule operations. 

The same "problem" occurs of course for other hardware devices such as
network, keyboard, mouse, where a transaction consists of more than a
single read or write to a single register. 

--
Mats
> 
> If you could gather some scheduler traces and send
> results, it will give us a good idea of what's going
> on and why. The multi-core support is new and not
> widely tested so it's possible that it is being
> overly aggressive or perhaps even buggy.
> 
> Emmanuel.
> 
> 
> On Fri, Dec 01, 2006 at 06:11:32PM +0800, Xu, Anthony wrote:
> > Emmanue,
> > 
> > I found that unnecessary VCPU migration happens again.
> > 
> > 
> > My environment is,
> > 
> > IPF two sockes, two cores per socket, 1 thread per core.
> > 
> > There are 4 core totally.
> > 
> > There are 3 domain, they are all UP,
> > So there are 3 VCPU totally.
> > 
> > One is domain0,
> > The other two are VTI-domain.
> > 
> > I found there are lots of migrations.
> > 
> > 
> > This is caused by below code segment in function csched_cpu_pick.
> > When I comments this code segment, there is no migration in above 
> > enviroment. 
> > 
> > 
> > 
> > I have a little analysis about this code.
> > 
> > This code handls multi-core and multi-thread, that's very good,
> > If two VCPUS running on LPs which belong to the same core, then the
> > performance
> > is bad, so if there are free LPS, we should let this two 
> VCPUS run on
> > different cores.
> > 
> > This code may work well with para-domain.
> > Because para-domain is seldom blocked,
> > It may be block due to guest call "halt" instruction.
> > This means if a idle VCPU is running on a LP,
> > there is no non-idle VCPU running on this LP.
> > In this evironment, I think below code should work well.
> > 
> > 
> > But in HVM environment, HVM is blocked by IO operation,
> > That is to say, if a idle VCPU is running on a LP, maybe a
> > HVM VCPU is blocked, and HVM VCPU will run on this LP, when
> > it is woken up.
> > In this evironment, below code cause unnecessary migrations.
> > I think this doesn't reach the goal ot this code segment.
> > 
> > In IPF side, migration is time-consuming, so it caused some 
> performance
> > degradation.
> > 
> > 
> > I have a proposal and it may be not good.
> > 
> > We can change the meaning of idle-LP,
> > 
> > Idle-LP means a idle-VCPU is running on this LP, and there 
> is no VCPU
> > blocked on this
> > LP.( if this VCPU is woken up, this VCPU will run on this LP).
> > 
> > 
> > 
> > --Anthony
> > 
> > 
> >         /*
> >          * In multi-core and multi-threaded CPUs, not all 
> idle execution
> >          * vehicles are equal!
> >          *
> >          * We give preference to the idle execution vehicle with the
> > most
> >          * idling neighbours in its grouping. This distributes work
> > across
> >          * distinct cores first and guarantees we don't do something
> > stupid
> >          * like run two VCPUs on co-hyperthreads while 
> there are idle
> > cores
> >          * or sockets.
> >          */
> >         while ( !cpus_empty(cpus) )
> >         {
> >             nxt = first_cpu(cpus);
> > 
> >             if ( csched_idler_compare(cpu, nxt) < 0 )
> >             {
> >                 cpu = nxt;
> >                 cpu_clear(nxt, cpus);
> >             }
> >             else if ( cpu_isset(cpu, cpu_core_map[nxt]) )
> >             {
> >                 cpus_andnot(cpus, cpus, cpu_sibling_map[nxt]);
> >             }
> >             else
> >             {
> >                 cpus_andnot(cpus, cpus, cpu_core_map[nxt]);
> >             }
> > 
> >             ASSERT( !cpu_isset(nxt, cpus) );
> >         }
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel
> 
> 
> 



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel