Re: [Xen-devel] xen/arm: Domain not fully destroyed when using credit2

[Julien Grall <julien.grall@xxxxxxx>]

On 25/01/17 12:40, Andrew Cooper wrote:
> On 25/01/17 12:38, Julien Grall wrote:
> > Hi Dario,
> >
> > On 25/01/17 11:10, Dario Faggioli wrote:
> > > On Tue, 2017-01-24 at 15:06 +0000, Julien Grall wrote:
> > > > On 24/01/17 14:16, Dario Faggioli wrote:
> > > > > There, we have tracing (BTW, did that made it to ARM eventually?)
> > > > > and
> > > > > there's TRC_PM_IDLE_ENTRY/EXIT which do pretty much the same of
> > > > > your
> > > > > printk-s.
> > > > There is patch on the ML for xentrace support (see [1]) but nothing
> > > > has
> > > > been upstreamed yet. Waiting for a new version from the contributor.
> > > Yep, that was I was remembering, and referring to. Thanks for the
> > > update.
> > >
> > > > > And if I look at it, I do see even totally idle (from the scheduler
> > > > > point of view) pCPUs, I indeed see them going back and forth from
> > > > > and
> > > > > to C3.
> > > > My knowledge on x86 is limited. When does a CPU decides to leave the
> > > > idle mode?
> > > I'm not an expert of that part either. Jan and Andrew for sure know
> > > best how monitor/mwait works (both in general, and our own
> > > implementation).
> > >
> > > What I know (and can quickly infer from glancing at the code), is that
> > > timers are certainly involved.
> > >
> > > In fact, we wake up when the most imminent timer would expire (see
> > > mwait_idle_with_hints()), and a timer set by the scheduler fully
> > > qualifies as being the one (if it's the most imminent).
> > >
> > > My point was that, still from scheduling perspective, neither Credit1
> > > nor Credit2 sets a wakeup timer for idle pCPUs.
> > >
> > > Well, in Credit1, the master_ticker timer is never stopped (while,
> > > e.g., the per-pCPU tick is stopped before entering deep sleep,
> > > via sched_tick_suspend(), see commit 964fae8ac), but that's only 1
> > > pCPU.
> > The function sched_tick_suspend is never called on ARM. The power
> > saving in Xen ARM is still very limited and this would need to be
> > updated in the future.
> >
> > So I guess that's why I still see interrupt coming on the idle pCPU
> > when credit1 is used. Looking at credit2, the callback tick_suspend is
> > not called. Does it mean there is no per-pCPU timer?
> >
> > Now, from my understanding, if we decide to call sched_tick_suspend on
> > ARM before idling. We will likely have the same problem with credit1
> > because there is no more interrupt to wake-up the pCPU.
> >
> > But I don't think this is an issue in the scheduler. IHMO, the problem
> > is in the RCU. Indeed a CPU in lower power mode (i.e  wfi on ARM or
> > pm_idle on x86 is been executed) will never get out to tell to the RCU
> > : "I am quiet, go ahead". So the RCU will never be able to reclaim the
> > memory and will result on a memory exhaustion if the pCPU never
> > receive an interrupt (this could happen if pCPU has never ran a guest).
> Yes.  This is a core problem, not ARM specific.
>
> x86 is saved by the time calibration rendezvous which IPIs all cores
> every 1s.
>
> > The question now, is how to fix it?
>
> This is going to involve a better understanding of how RCU is supposed
> to work.
I think we all agree that someone needs to kick the other pCPU to check 
whether the RCU is been used.
Looking at the documentation of our RCU code ([1]), section "RCU 
Implementations" it seems that we are expecting a timer to kick 
periodically pCPU and if there is some RCU work pending.
[1] http://lse.sourceforge.net/locking/rcupdate.html

> ~Andrew
--
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel