Re: [Xen-devel] Strange interdependace between domains

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

On gio, 2014-02-13 at 22:25 +0000, Simon Martin wrote:
> Thanks for all the replies guys.
> 
:-)

> Don> How many instruction per second a thread gets does depend on the
> Don> "idleness" of other threads (no longer just the hyperThread's
> Don> parther).
> 
> This    seems    a    bit    strange   to   me. In my case I have time
> critical  PV  running  by  itself  in a CPU pool. So Xen should not be
> scheduling it, so I can't see how this Hypervisor thread would be affected.
> 
I think Don is referring to the idleness of the other _hardware_ threads
in the chip, rather than software threads of execution, either in Xen or
in Dom0/DomU. I checked his original e-mail and, AFAIUI, he seems to
confirm that the throughput you get on, say, core 3, depends on what
it's sibling core (which really is his sibling hyperthread, again in the
hardware sense... Gah, the terminology is just a mess! :-P). He seems to
also add the fact that there is a similar kind of inter-dependency
between all the hardware hyperthread, not just between siblings.

Does this make sense Don?

> >> 6.- All VCPUs are pinned:
> >> 
> Dario> Right, although, if you use cpupools, and if I've understood what 
> you're
> Dario> up to, you really should not require pinning. I mean, the isolation
> Dario> between the RT-ish domain and the rest of the world should be already 
> in
> Dario> place thanks to cpupools.
> 
> This  is what I thought, however when running looking at the vcpu-list
> I  CPU  affinity  was "all" until I starting pinning. As I wasn't sure
> whether  that  was  "all  inside this cpu pool" or "all" I felt it was
> safer to do it explicitly.
> 
Actually, you are right, we could put things in a way that results more
clear, when one observes the output! So, I confirm that, despite the
fact that you see "all", that all is relative to the cpupool the domain
is assigned to.

I'll try to think on how to make this more evident... A note in the
manpage and/or the various sources of documentation, is the easy (but
still necessary, I agree) part, and I'll add this to my TODO list.
Actually modifying the output is more tricky, as affinity and cpupools
are orthogonal by design, and that is the right (IMHO) thing.

I guess trying to tweak the printf()-s in `xl vcpu-list' would not be
that hard... I'll have a look and see if I can come up with a proposal.

> Dario> So, if you ask me, you're restricting too much things in
> Dario> pool-0, where dom0 and the Windows VM runs. In fact, is there a
> Dario> specific reason why you need all their vcpus to be statically
> Dario> pinned each one to only one pcpu? If not, I'd leave them a
> Dario> little bit more of freedom.
> 
> I agree with you here, however when I don't pin CPU affinity is "all".
> Is this "all in the CPU pool"? I couldn't find that info.
> 
Again, yes: once a domain is in a cpupool, no matter what its affinity
says, it won't ever reach a pcpu assigned to another cpupool. The
technical reason is that each cpupool is ruled by it's own (copy of a)
scheduler, even if you use, e.g., credit, for both/all the pools. In
that case, what you will get are two full instances of credit,
completely independent between each other, each one in charge only of a
very specific subset of pcpus (as mandated by cpupools). So, different
runqueues, different data structures, different anything.

> Dario> What I'd try is:
> Dario>  1. all dom0 and win7 vcpus free, so no pinning in pool0.
> Dario>  2. pinning as follows:
> Dario>      * all vcpus of win7 --> pcpus 1,2
> Dario>      * all vcpus of dom0 --> no pinning
> Dario>    this way, what you get is the following: win7 could suffer 
> sometimes,
> Dario>    if all its 3 vcpus gets busy, but that, I think is acceptable, at
> Dario>    least up to a certain extent, is that the case?
> Dario>    At the same time, you
> Dario>    are making sure dom0 always has a chance to run, as pcpu#0 would be
> Dario>    his exclusive playground, in case someone, including your pv499
> Dario>    domain, needs its services.
> 
> This  is  what  I  had when I started :-). Thanks for the confirmation
> that I was doing it right. However if the hyperthreading is the issue,
> then I will only have 2 PCPU available, and I will assign them both to
> dom0 and win7.
> 
Yes, with hyperthreading in mind, that is what you should do.

Once we will have confirmed that hyperthreading is the issue, we'll see
what we can do. I mean, if, in your case, it's fine to 'waste' a cpu,
then ok, but I think we need a general solution for this... Perhaps with
a little worse performances than just leaving one core/hyperthread
completely idle, but at the same time more resource efficient.

I wonder how tweaking the sched_smt_power_savings would deal with
this...

> Dario> Right. Are you familiar with tracing what happens inside Xen
> Dario> with xentrace and, perhaps, xenalyze? It takes a bit of time to
> Dario> get used to it but, once you dominate it, it is a good mean for
> Dario> getting out really useful info!
> 
> Dario> There is a blog post about that here:
> Dario> 
> http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze/
> Dario> and it should have most of the info, or the links to where to
> Dario> find them.
> 
> Thanks for this. If this problem is more than the hyperthreading then
> I will definitely use it. Also looks like it might be useful when I
> start looking at the jitter on the singleshot timer (which should be
> in a couple of weeks).
> 
It will reveal to be very useful for that, I'm sure! :-)

Let us know how the re-testing goes.

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Attachment: signature.asc
Description: This is a digitally signed message part

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel