Re: [Xen-devel] Re: Fine-grained proxy resource charging

[John L Griffin <jlg@xxxxxxxxxx>]

Howdy Lucy, and thanks for the detailed reply.

> I/O bandwidth is a natural extension for resource accounting and can
> be addressed as well.

Agreed.  An interesting extension of this example: what if a non-I/O 
privileged service domain B (say, a domain that provides a object 
encryption service) must sometimes perform some network I/O through a 
third domain (say, downloading a new encryption algorithm from an external 
source) in order to continue servicing A's requests?  In this case, it 
will need to be possible for B to transitively specify on whose behalf the 
third-party work is being performed.

> We believe that the amount of memory page exchanges (between A and B,
> C and B, etc.) is a relatively accurate "hint" for splitting the CPU
> overhead in B with respect to A, C, etc.

I do like the simplicity of this approach -- in fact, I had exactly the 
same thought about using grant table operations to approximate proxy CPU 
consumption.  (Imagine my surprise when I opened up Login magazine and 
found the summary of the Usenix talks...)

However, I'd already rejected the idea, and decided to concentrate on 
alternate/finer-grained/internal-to-B approaches, for several reasons:

- Preventing DoS.  The count-page-exchanges scheme is good for CPU 
accounting, especially since it can be done with an unmodified B. However, 
in the context of actively rate-limiting A's resource consumption (perhaps 
by delaying or failing the page exchanges) it wouldn't work, since by the 
time B invokes a grant table operation it's too late -- B has already been 
affected.

- Non-correlation of grant table operations to resource consumption.  I 
don't have any empirical evidence for this, but I envision "service 
domain" scenarios like the one above, where there may be very few page 
exchanges and yet wildly different amounts of work performed. 

- Overcharging (as in my previous message).  I don't have any evidence for 
this either, but my first thought was that it might be an overpessimistic 
assumption to allocate 100% of B's resource usage to its clients.

- Transitivity (as in my example above).

Regardless, I agree with Rob's earlier point -- even a coarse-grained 
solution is better than nothing.  Perhaps the most interesting result from 
your work is just how substantial B's proxy CPU usage can be (and its 
effect on provisioning a system), and how we/the Xen community should 
perhaps focus on making that more efficient.

> The problem gets
> much harder and more complex when there are different drivers hosted
> by the same driver domain.

Could anybody comment on the current status of breaking dom0 into multiple 
single-function service domains, and/or not having a driver domain hosting 
multiple drivers?  I'm not caught up on Xen's current events, pun 
intended.

> Yes, we are also looking at how this overhead can be taken into
> account during the CPU scheduling for making a smarter resource
> allocation decision. The trade off here seems to be how one can
> enforce such decisions: either via a new scheduling policy (requires
> changes to Xen) or via changing the next period resource allocation
> from the outside of Xen based on the previous usage (one can use xm
> bvt .... or xm sedf facility for changing the allocation). It might
> depend on the targeted granularity of resource allocation decisions.

Actually, my thought was the scheduling that happens inside the domain 
(such as B's selection of which network packet it will process next).  I'd 
started to brainstorm about doing different scheduling at the 
Xen-scheduling level, but couldn't come up with any ideas that didn't have 
the potential of adversely affecting non-A domains.  Maybe it'd be 
possible to split a driver domain into a group of cooperating 
mini-domains, that collectively accomplish the same purpose but are 
independently scheduled?  Each mini-domain would service exactly one 
A-type domain.  (How bad are context switches in ring 1...maybe not too 
bad, so the main challenge would be architecting the mini-domains.)  I'd 
be happy to join in (from a distance) on any continuing whiteboard 
discussions you have along these lines.

> The other interesting question here is to provide some kind of
> performance isolation: for example, limiting the impact of the
> excessive traffic to one domain (say A) and its related overhead in
> driver domain (B)  on performance of the other domains.

Agreed!  This is exactly along the lines of what I was thinking about 
having Xen expose A's resource usage counts to B -- with the hope of 
allowing B (by scheduling or some other mechanism) to cut A off for 
awhile.

JLG


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel