Re: [Xen-devel] [PATCH-for-4.13] x86/mm: don't needlessly veto migration

[Paul Durrant <pdurrant@xxxxxxxxx>]

On Tue, 1 Oct 2019 at 14:06, George Dunlap <george.dunlap@xxxxxxxxxx> wrote:
>
[snip]
> >>>> But the question is, do we want the toolstack to have to become an
> >>>> expert in what hardware might have external dirty tracking, and whether
> >>>> such tracking is active?  At the moment that would mean either 1)
> >>>> putting that information inside of libxc, or 2) duplicating it across
> >>>> xapi and libxl, for instance.
> >>>
> >>> Why not? The toolstack is in charge of migration so why can't it decide 
> >>> whether it is 'safe' or not?
> >>
> >> First of all, it's not about what the toolstack can decide; it's what it
> >> knows.  It doesn't currently know anything about the details of devices
> >> themselves or how they relate to other functionality, such as migration.
> >
> > Doesn't it? Why?
> [snip]
> > It is, but I don't see that Xen should have any right of veto over
> > what a paticular toolstack wishes to do with the domains it has
> > created.
>
> I feel like the temperature of this conversation is really high, and I
> can't really figure out why.  Could I ask that we try to turn down the
> heat a bit, and perhaps help Jan and I figure out where you're coming from?
>

It is not my intention it appears this way, but text-based
communication is not great in this kind of case.

I'll explain the background, whilst trying to keep proprietary stuff
proprietary... We (Citrix) have some SR-IOV hardware where VFs are
passed through to a domain via a dedicated emulator. The PF can
potentially provide information about which guest pages have been
dirtied by DMA, and thus it is feasible to migrate domains with VFs
assigned. However, current development is frustrated by Xen's refusal
to enable logdirty and so we now have a patch (a prior version of the
one I posted) that rectifies this. I would now like to get this
upstream as I do not see that any harm can be caused to the system by
allowing a domain with IOMMU mappings to have logdirty enabled, unless
the P2M is shared.

> > It can, in the face of an arbitrary device, use an
> > emulator such as QEMU to deal with the pass-through and having so
> > decided knows that it can't get dirty page information, and hence the
> > domain cannot be safely migrated. In the face of a device it knows
> > about though e.g. a GPU, it can run a dedicated emulator from which it
> > can get dirty page information and hence (providing shared EPT is not
> > in use) it knows the domain can be migrated.
>
> xapi knows what devices *it has asked Xen to pass through*.  Xen knows
> *what devices it gave to the guest*.

I don't understand the difference.

>
> xapi can *gather* specific information about devices (topology,
> characteristics, &c) from Linux and Xen; Xen has it already.
>
> It seems you've encoded in xapi information about how Xen is
> implemented.  That could change: More features could begin to interact
> with logdirty, or with devices which can implement their own
> logdirty-like functionality.  If/when those changes happen, we can
> update the rules for when logdirty works within the patch series itself
> in Xen.  If you instead encode that knowlegde in xapi, then xapi needs
> to be updated to keep in sync with the internal implementation of the
> hypervisor.
>

I don't follow. I don't think we have encoded information about how
Xen works. What we have are buch of things involved in migration...
Xen is one, and all the others are emulators. So, we enable logdirty
in Xen and then ask it what pages the guest has touched, and we enable
dirty tracking in the emulators and ask them what pages they (or the
h/w they manage) as touched. That info. feeds into a combined dirty
page list and that then informs the migration stream.

> In any case, having emulators which can handle logdirty externally
> report their own capability seems a much better way of doing things than
> hard-coding in xapi which emulators know how to do what.

In the long run, yes, but I don't see why Xen should be making the
assumption that no current emulator can do that. It seems entirely the
wrong place to be doing that. At some level Xen is 'just another
emulator' and ought not to care what its peers are capable of.

>
> >> Secondly, you haven't answered the question about duplication.  Where do
> >> you propose to put this functionality?
> >>
> >
> > Different toolstacks can have different capabilities. If libxl is
> > unaware of a devices capability to provide dirty page information, but
> > XAPI is aware, then why is that a problem?
>
> So it sounds like you've already done a lot of this work in xapi.  But
> that only benefits XenServer and derivatives: all of Citrix's other
> partners who want to do something similar will have to completely
> duplicate all of that functionality.  It should be obvious why that's
> sub-optimal.

The changes in XAPI are not vast; the main complexity is in the device
emulator (to provide information during the live phase of migration)
but I still don't see why Citrix's choice of closed vs. open source
implementation of the emulator really has anything to do with this. It
is still my opinion that Xen's only valid reason for refusing to
enable logdirty for a domain is one of host safety and I still haven't
heard an argument as to why Xen *is* right to refuse in other
circumstances.

  Paul

>
>  -George

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel