Re: [Xen-devel] [RFC PATCH 3/7] xen: psr: reserve an RMID for each core

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

On Wed, 2015-04-08 at 14:28 +0100, George Dunlap wrote:
> On 04/04/2015 03:14 AM, Dario Faggioli wrote:
> > This allows for a new item to be passed as part of the psr=
> > boot option: "percpu_cmt". If that is specified, Xen tries,
> > at boot time, to associate an RMID to each core.
> > 
> > XXX This all looks rather straightforward, if it weren't
> >     for the fact that it is, apparently, more common than
> >     I though to run out of RMID. For example, on a dev box
> >     we have in Cambridge, there are 144 pCPUs and only 71
> >     RMIDs.
> 
> Is that because you have 2 sockets?
> 
It has 4 sockets. Chao explained in one of his mails that there usually
is 2 or more RMIDs per hardware thread.

> There's no need to keep RMIDs unique across sockets, is there?  E.g.,
> socket 0 cpu 0 and socket 1 cpu 0 can have the same RMID, because cache
> and the MSRs are per-socket.
> 
Exactly. And in fact, I just added to my TODO list improving Xen's
current PSR support to take per-socketness of RMIDs correctly into
account.

> If we're doing things on a per-domain basis, having the same RMID
> allocated for each socket sort of makes sense; but even then, if you
> know a domain is only going to run on a given socket, there's no reason
> in theory we couldn't use same RMID for a different domain on the other
> socket (assuming it was only going to run on the other socket).
> 
Right now CMT is per-domain and, on a box with 71 available RMIDs
_per_each_socket_, we have (in Xen) an array of 71 possible RMIDs (72,
but RMID 0 is treated specially) to be assigned to domains.

Independently on where a domain will ever run, we can use separate
arrays, and each domain can have an RMID on each socket. If, as you say,
there are well known restrictions, a domain can avoid having RMIDs for
certain sockets. In the worst case (all domains can run everywhere), we
use the same amount of RMIDs, in better/best cases, we use less of them.

This is even more important when looking at per-pCPU CMT configurations.
In fact, right now, on that box, I have more than 71 pCPUs, so I can't
associate an RMID to all the pCPUs. However, with the proper per-socket
support implemented, I not only will be able to associate an RMID to all
the pCPUs, but I'll have 35 RMIDs on each socket free. :-)

> One advantage of doing things of a per-vcpu level is that you wouldn't
> have to worry about inter-socket RMID issues.
>
Sorry, but I'm lost again, I'm afraid. What do you mean with this?

Thanks and Regards,
Dario

Attachment: signature.asc
Description: This is a digitally signed message part

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel