Re: [Xen-devel] PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross <jgross@xxxxxxxx>]

On 07/24/2015 05:58 PM, Dario Faggioli wrote:
> On Fri, 2015-07-24 at 17:24 +0200, Juergen Gross wrote:
> > On 07/24/2015 05:14 PM, Juergen Gross wrote:
> > > On 07/24/2015 04:44 PM, Dario Faggioli wrote:
> > > > In fact, I think that it is the topology, i.e., what comes from MSRs,
> > > > that needs to adapt, and follow vNUMA, as much as possible. Do we agree
> > > > on this?
> > > I think we have to be very careful here. I see two possible scenarios:
> > >
> > > 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will
> > >      try to schedule the vcpus according to their numa affinity. So they
> > >      can change pcpus at any time in case of very busy guests. I don't
> > >      think the linux kernel should treat the cpus differently in this
> > >      case as it will be in vane regarding the Xen scheduler's activity.
> > >      So we should use the "null" topology in this case.
> > Sorry, the topology should reflect the vcpu<->numa-node relations, of
> > course, but nothing else (so flat topolgy in each numa node).
> Yeah, I was replying to this point saying something like this right
> now... Luckily, I've seen this email! :-P
>
> With this semantic, I fully agree with this.
>
> > > 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen
> > >      scheduler can't move vcpus between pcpus, so the linux kernel should
> > >      see the real topology of the used pcpus in order to optimize for this
> > >      picture.
> Mmm... I did think about this too, but I'm not sure. I see the value of
> this of course, and the reason why it makes sense. However, pinning can
> change on-line, via `xl vcpu-pin' and stuff. Also migration could make
> things less certain, I think. What happens if we build on top of the
> initial pinning, and then things change?
If we can fiddle with the masks on boot, we could do it in a running
system, too. Another advantage with not relying on cpuid. :-)

> To be fair, there is stuff building on top of the initial pinning
> already, e.g., from which physical NUMA node we allocate the memory
> relies depends exactly on that. That being said, I'm not sure I'm
> comfortable with adding more of this...
>
> Perhaps introduce an 'immutable_pinning' flag, which will prevent
> affinity to be changed, and then bind the topology to pinning only if
> that one is set?
Hmm, this would require disabling migration as well. Or enable it
with "--force" only.

> > > > Maybe, there is room for "fixing" this at this level, hooking up inside
> > > > the scheduler code... but I'm shooting in the dark, without having check
> > > > whether and how this could be really feasible, should I?
> > > Uuh, I don't think a change of the scheduler on behalf of Xen is really
> > > appreciated. :-)
> I'm sure it would (have been! :-)) a true and giant nightmare!! :-D
>
> > > > One thing I don't like about this approach is that it would potentially
> > > > solve vNUMA and other scheduling anomalies, but...
> > > >
> > > > > cpuid instruction is available for user mode as well.
> > > > ...it would not do any good for other subsystems, and user level code
> > > > and apps.
> > > Indeed. I think the optimal solution would be two-fold: give the
> > > scheduler the information it is needing to react correctly via a
> > > kernel patch not relying on cpuid values and fiddle with the cpuid
> > > values from xen tools according to any needs of other subsystems and/or
> > > user code (e.g. licensing).
> So, just to check if I'm understanding is correct: you'd like to add an
> abstraction layer, in Linux, like in generic (or, perhaps, scheduling)
> code, to hide the direct interaction with CPUID.
> Such layer, on baremetal, would just read CPUID while, on PV-ops, it'd
> check with Xen/match vNUMA/whatever... Is this that you are saying?
Sort of, yes.

I just wouldn't add it, as it is already existing (more or less). It
can deal right now with AMD and Intel, we would "just" have to add Xen.

> If yes, I think I like it...
I hope e.g. the KVM guys will like it, too. :-)


Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel