Re: [Xen-devel] [RFC PATCH 21/24] ARM: vITS: handle INVALL command

[Julien Grall <julien.grall@xxxxxxx>]

Hi Stefano,

On 07/12/16 20:20, Stefano Stabellini wrote:
> On Tue, 6 Dec 2016, Julien Grall wrote:
> > On 06/12/2016 22:01, Stefano Stabellini wrote:
> > > On Tue, 6 Dec 2016, Stefano Stabellini wrote:
> > > > moving a vCPU with interrupts assigned to it is slower than moving a
> > > > vCPU without interrupts assigned to it. You could say that the
> > > > slowness is directly proportional do the number of interrupts assigned
> > > > to the vCPU.
> > > To be pedantic, by "assigned" I mean that a physical interrupt is routed
> > > to a given pCPU and is set to be forwarded to a guest vCPU running on it
> > > by the _IRQ_GUEST flag. The guest could be dom0. Upon receiving one of
> > > these physical interrupts, a corresponding virtual interrupt (could be a
> > > different irq) will be injected into the guest vCPU.
> > >
> > > When the vCPU is migrated to a new pCPU, the physical interrupts that
> > > are configured to be injected as virtual interrupts into the vCPU, are
> > > migrated with it. The physical interrupt migration has a cost. However,
> > > receiving physical interrupts on the wrong pCPU has an higher cost.
> > I don't understand why it is a problem for you to receive the first interrupt
> > to the wrong pCPU and moving it if necessary.
> >
> > While this may have an higher cost (I don't believe so) on the first received
> > interrupt, migrating thousands of interrupts at the same time is very
> > expensive and will likely get Xen stuck for a while (think about ITS with a
> > single command queue).
> >
> > Furthermore, the current approach will move every single interrupt routed a
> > the vCPU, even those disabled. That's pointless and a waste of resource. You
> > may argue that we can skip the ones disabled, but in that case what would be
> > the benefits to migrate the IRQs while migrate the vCPUs?
> >
> > So I would suggest to spread it over the time. This also means less headache
> > for the scheduler developers.
> The most important aspect of interrupts handling in Xen is latency,
> measured as the time between Xen receiving a physical interrupt and the
> guest receiving it. This latency should be both small and deterministic.
>
> We all agree so far, right?
>
>
> The issue with spreading interrupts migrations over time is that it makes
> interrupt latency less deterministic. It is OK, in the uncommon case of
> vCPU migration with interrupts, to take a hit for a short time.  This
> "hit" can be measured. It can be known. If your workload cannot tolerate
> it, vCPUs can be pinned. It should be a rare event anyway.  On the other
> hand, by spreading interrupts migrations, we make it harder to predict
> latency. Aside from determinism, another problem with this approach is
> that it ensures that every interrupt assigned to a vCPU will first hit
> the wrong pCPU, then it will be moved.  It guarantees the worst-case
> scenario for interrupt latency for the vCPU that has been moved. If we
> migrated all interrupts as soon as possible, we would minimize the
> amount of interrupts delivered to the wrong pCPU. Most interrupts would
> be delivered to the new pCPU right away, reducing interrupt latency.
Migrating all the interrupts can be really expensive because in the 
current state we have to go through every single interrupt and check 
whether the interrupt has been routed to this vCPU. We will also route 
disabled interrupt. And this seems really pointless. This may need some 
optimization here.
With ITS, we may have thousand of interrupts routed to a vCPU. This 
means that for every interrupt we have to issue a command in the host 
ITS queue. You will likely fill up the command queue and add much more 
latency.
Even if you consider the vCPU migration to be a rare case. You could 
still get the pCPU stuck for tens of milliseconds, the time to migrate 
everything. And I don't think this is not acceptable.
Anyway, I would like to see measurement in both situation before 
deciding when LPIs will be migrated.
> Regardless of how we implement interrupts migrations on ARM, I think it
> still makes sense for the scheduler to know about it. I realize that
> this is a separate point. Even if we spread interrupts migrations over
> time, it still has a cost, in terms of latency as I wrote above, but also
> in terms of interactions with interrupt controllers and ITSes. A vCPU
> with no interrupts assigned to it poses no such problems. The scheduler
> should be aware of the difference. If the scheduler knew, I bet that
> vCPU migration would be a rare event for vCPUs that have many interrupts
> assigned to them. For example, Dom0 vCPU0 would never be moved, and
> dom0_pin_vcpus would be superfluous.
The number of interrupts routed to a vCPU will vary over the time, this 
will depend what the guest decides to do, so you need scheduler to 
adapt. And in fine, you give the guest a chance to "control" the 
scheduler depending how the interrupts are spread between vCPU.
If the number increases, you may end up to have the scheduler to decide 
to not migrate the vCPU because it will be too expensive. But you may 
have a situation where migrating a vCPU with many interrupts is the only 
possible choice and you will slow down the platform.
Cheers,

--
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel