[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] NUMA-aware VM placement in Xen

On 24/02/12 10:12, Dario Faggioli wrote:
Hi guys,

As some of you know I'm working on putting some kind of NUMA-aware
placement of the various VMs within Xen. This means I'm investigating
deeply how memory allocation works, which isn't an easy task for me (I
started completely from scratch), so forgive me if I say something
wrong! :-P

The status is I'm dealing for a while with a "design issue" I'd be very
glad to discuss with someone, as I'm not sure which path to go for...

To keep it short, what I need is a place --ideally during VM creation--
where I can check how much memory a VM wants against how much memory is
available in the various NUMA-nodes, and use this as the basis of my
decision. The question is, where is this place?

I traced memory related calls (e.g., for HVMs) from libxl__build_hvm to
xc_hvm_build_target_mem to setup_guest to xc_domain_populate_physmap and
alloc_domheap_pages. The last twos have been my target for a while, but
I'm not so sure they would be the right choice, mainly because both of
them are called for allocating _only_part_ of the VM's memory, i.e.,
some extents of it at each call (am I right?).

Basically, given alloc_domheap_pages uses d->node_affinity for deciding
from which node(s) to actually take memory from, I was planning to
either use the same mask or build a new one with similar purposes, the
problem being _where_ to populate it with the proper nodes.
I'm now looking at xc_domain_setmaxmem-->do_domctl(XEN_DOMCTL_max_mem),
although I think it's too early, and I'd end up guessing wrt a lot of
aspects... But considering xm/xend was doing the same even earlier (at
least I think)...
So the first question is, where should the decision about NUMA placement be made, and the second is how that level should implement it.

Doing it at the libxc level I think is not right. It seems to me we have two options: * Have libxl do the NUMA placement on behalf of the toolstack. In that case, the libxl_domain_create_new function should look at the available memory, the NUMA layout, &c, and then set d->node_affinity before calling xc_hvm_build. * Have the toolstack do it. In this case, you'd be modifying xl to set d->node_affinity before calling libxl's domain creation function.

Do those options work?  Let me know if I've misunderstood anything.

Any thoughts one way or the other from anyone?

I'd be tempted to have it be optional -- you can set "numa=auto" and the domain creation function will do the simple thing; or you can set "numa=manual" and have the toolstack / config file set the nodes manually. That would translate pretty well to config files as well -- more "set the knobs" administrators could set the numa layout in the config file manually if they wanted.



Sorry fro writing so much... Any help/ideas you feel comfortable with
sharing would be very appreciated! :-)

Thanks a lot and regards,

Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.