Re: [Xen-devel] [RFC 16/16] xen/arm: Track page accessed between batch of Set/Way operations

[Julien Grall <julien.grall@xxxxxxx>]

On 06/11/2018 17:43, Stefano Stabellini wrote:
> On Mon, 5 Nov 2018, Julien Grall wrote:
> > Hi Stefano,
> >
> > On 11/5/18 9:35 PM, Stefano Stabellini wrote:
> > > On Mon, 8 Oct 2018, Julien Grall wrote:
> > > > At the moment, the implementation of Set/Way operations will go through
> > > > all the entries of the guest P2M and flush them. However, this is very
> > > > expensive and may render unusable a guest OS using them.
> > > >
> > > > For instance, Linux 32-bit will use Set/Way operations during secondary
> > > > CPU bring-up. As the implementation is really expensive, it may be
> > > > possible
> > > > to hit the CPU bring-up timeout.
> > > >
> > > > To limit the Set/Way impact, we track what pages has been of the guest
> > > > has been accessed between batch of Set/Way operations. This is done
> > > > using bit[0] (aka valid bit) of the P2M entry.
> > > This is going to improve performance of ill-mannered guests at the cost
> > > of hurting performance of well-mannered guests. Is it really a good
> > > trade-off? Should this behavior at least be configurable with a Xen
> > > command line?
> > Well, we have the choice between not been able to boot Linux 32-bit anymore or
> > have a slight impact at the boot time for all guests.
> Wait -- I thought that with the set/way emulation introduced by patch
> #15 we would be able to boot Linux 32-bit already. This patch is a
> performance improvement. Or is it actually needed to boot Linux 32-bit?
The problem is Linux 32-bit calls a few time set/way during secondary CPU bring 
up. It also has a timeout of 1s to fully boot that CPU. In my testing, I can 
easily hit the timeout even with a small amount of memory.
If we don't start tracking the page from the beginning, then you would need to 
clean the full RAM the first time. If you start to track from the beginning, you 
may just have to clean a couple of MB.
> > As you may have noticed the command line is been suggested below. I didn't yet
> > implemented as we agreed at Connect it would be good to start getting feedback
> > on it.
> Sure. I was thinking about this -- does it make sense to have different
> defaults for 32bit and 64bit guests?
>
> 32bit -> default p2m_invalidate_root
> 64bit -> default not
The decision is not that easy. While Linux arm64 does not contain set/way 
anymore, UEFI still use them (I checked it a couple of months ago).
I haven't done any benchmark yet. That's my next step.

> > I am not entirely to understand your last sentence, this feature is turned off
> > when an IOMMU is present. So what is your use case here?
>   
> My sentence was badly written, sorry. I meant to say that even when an
> IOMMU is NOT present, there are cases where we might not want to call
> p2m_invalidate_root.
Before implementing a command line option (or even plumbing that to the guest 
configuration), I want to understand what is the real impact on the boot time 
for "normal" guest.
Cheers,

--
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel