Re: [Xen-devel] [PATCH-for-4.13] x86/mm: don't needlessly veto migration

[Paul Durrant <pdurrant@xxxxxxxxx>]

On Tue, 1 Oct 2019 at 12:09, George Dunlap <george.dunlap@xxxxxxxxxx> wrote:
>
> On 10/1/19 11:40 AM, Paul Durrant wrote:
> >  -----Original Message-----
> >> From: George Dunlap <george.dunlap@xxxxxxxxxx>
> >> Sent: 01 October 2019 11:34
> >> To: Paul Durrant <Paul.Durrant@xxxxxxxxxx>; 'Jan Beulich' 
> >> <jbeulich@xxxxxxxx>
> >> Cc: Andrew Cooper <Andrew.Cooper3@xxxxxxxxxx>; Roger Pau Monne 
> >> <roger.pau@xxxxxxxxxx>; xen-
> >> devel@xxxxxxxxxxxxxxxxxxxx; Juergen Gross <jgross@xxxxxxxx>; Wei Liu 
> >> <wl@xxxxxxx>
> >> Subject: Re: [Xen-devel] [PATCH-for-4.13] x86/mm: don't needlessly veto 
> >> migration
> >>
> >> On 10/1/19 11:24 AM, Paul Durrant wrote:
> >>>> -----Original Message-----
> >>>> From: Jan Beulich <jbeulich@xxxxxxxx>
> >>>> Sent: 01 October 2019 11:15
> >>>> To: Paul Durrant <Paul.Durrant@xxxxxxxxxx>
> >>>> Cc: Andrew Cooper <Andrew.Cooper3@xxxxxxxxxx>; George Dunlap 
> >>>> <George.Dunlap@xxxxxxxxxx>; Roger Pau
> >>>> Monne <roger.pau@xxxxxxxxxx>; xen-devel@xxxxxxxxxxxxxxxxxxxx; Juergen 
> >>>> Gross <jgross@xxxxxxxx>; Wei
> >> Liu
> >>>> <wl@xxxxxxx>
> >>>> Subject: Re: [Xen-devel] [PATCH-for-4.13] x86/mm: don't needlessly veto 
> >>>> migration
> >>>>
> >>>> On 01.10.2019 11:36, Paul Durrant wrote:
> >>>>>> -----Original Message-----
> >>>>>> From: Jan Beulich <jbeulich@xxxxxxxx>
> >>>>>> Sent: 01 October 2019 10:19
> >>>>>> To: Paul Durrant <Paul.Durrant@xxxxxxxxxx>
> >>>>>> Cc: Andrew Cooper <Andrew.Cooper3@xxxxxxxxxx>; George Dunlap 
> >>>>>> <George.Dunlap@xxxxxxxxxx>; Roger
> >> Pau
> >>>>>> Monne <roger.pau@xxxxxxxxxx>; xen-devel@xxxxxxxxxxxxxxxxxxxx; Juergen 
> >>>>>> Gross <jgross@xxxxxxxx>;
> >> Wei
> >>>> Liu
> >>>>>> <wl@xxxxxxx>
> >>>>>> Subject: Re: [Xen-devel] [PATCH-for-4.13] x86/mm: don't needlessly 
> >>>>>> veto migration
> >>>>>>
> >>>>>> On 01.10.2019 10:52, Paul Durrant wrote:
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Jan Beulich <jbeulich@xxxxxxxx>
> >>>>>>>> Sent: 01 October 2019 09:46
> >>>>>>>> To: Paul Durrant <Paul.Durrant@xxxxxxxxxx>
> >>>>>>>> Cc: xen-devel@xxxxxxxxxxxxxxxxxxxx; Andrew Cooper 
> >>>>>>>> <Andrew.Cooper3@xxxxxxxxxx>; Roger Pau Monne
> >>>>>>>> <roger.pau@xxxxxxxxxx>; George Dunlap <George.Dunlap@xxxxxxxxxx>; 
> >>>>>>>> Juergen Gross
> >>>> <jgross@xxxxxxxx>;
> >>>>>> Wei
> >>>>>>>> Liu <wl@xxxxxxx>
> >>>>>>>> Subject: Re: [Xen-devel] [PATCH-for-4.13] x86/mm: don't needlessly 
> >>>>>>>> veto migration
> >>>>>>>>
> >>>>>>>> On 01.10.2019 10:28, Paul Durrant wrote:
> >>>>>>>>> Now that xl.cfg has an option to explicitly enable IOMMU mappings 
> >>>>>>>>> for a
> >>>>>>>>> domain, migration may be needlessly vetoed due to the check of
> >>>>>>>>> is_iommu_enabled() in paging_log_dirty_enable().
> >>>>>>>>> There is actually no need to prevent logdirty from being enabled 
> >>>>>>>>> unless
> >>>>>>>>> devices are assigned to a domain and that domain is sharing HAP 
> >>>>>>>>> mappings
> >>>>>>>>> with the IOMMU (in which case disabling write permissions in the 
> >>>>>>>>> P2M may
> >>>>>>>>> cause DMA faults).
> >>>>>>>>
> >>>>>>>> But that's taking into account only half of the reason of the
> >>>>>>>> exclusion. The other half is that assigned devices may cause pages
> >>>>>>>> to be dirtied behind the back of the log-dirty logic.
> >>>>>>>
> >>>>>>> But that's no reason to veto logdirty. Some devices have drivers (in 
> >>>>>>> dom0)
> >>>>>>> which can extract DMA dirtying information and set dirty tracking
> >>>>>>> information appropriately.
> >>>>>>
> >>>>>> It still needs a positive identification then: Such drivers should tell
> >>>>>> Xen for which specific devices such information is going to be 
> >>>>>> provided.
> >>>>>
> >>>>> Why does the hypervisor need have the right of veto though? Surely it is
> >>>>> the toolstack that should decide whether a VM is migratable in the
> >>>>> presence of assigned h/w. Xen need only be concerned with the integrity
> >>>>> of the host, which is why the check for ETP sharing remains.
> >>>>
> >>>> While the tool stack is to decide, the hypervisor is expected to 
> >>>> guarantee
> >>>> correct data coming back from XEN_DOMCTL_SHADOW_OP_{PEEK,CLEAN}.
> >>>
> >>> For some definition of 'correct', yes, and I don't think that this change 
> >>> violates any definition I
> >> can find in the domctl header.
> >>>
> >>> Note: there are already emulators that will be playing with the dirty map 
> >>> on an arbitrary and
> >> unsynchronized basis because they are emulating bus mastering h/w.
> >>
> >> But the question is, do we want the toolstack to have to become an
> >> expert in what hardware might have external dirty tracking, and whether
> >> such tracking is active?  At the moment that would mean either 1)
> >> putting that information inside of libxc, or 2) duplicating it across
> >> xapi and libxl, for instance.
> >
> > Why not? The toolstack is in charge of migration so why can't it decide 
> > whether it is 'safe' or not?
>
> First of all, it's not about what the toolstack can decide; it's what it
> knows.  It doesn't currently know anything about the details of devices
> themselves or how they relate to other functionality, such as migration.

Doesn't it? Why? It can, in the face of an arbitrary device, use an
emulator such as QEMU to deal with the pass-through and having so
decided knows that it can't get dirty page information, and hence the
domain cannot be safely migrated. In the face of a device it knows
about though e.g. a GPU, it can run a dedicated emulator from which it
can get dirty page information and hence (providing shared EPT is not
in use) it knows the domain can be migrated.

>  Xen knows about specific devices, and could (for instance) special-case
> specific device IDs or what-not; and Xen knows that device pass-through
> interacts with logdirty.  You're proposing to teach the toolstack about
> all of that.
>

Indeed, because it is in a much better position to know that kind of
information and indeed XAPI already does know that kind of
informaiton.

> Secondly, you haven't answered the question about duplication.  Where do
> you propose to put this functionality?
>

Different toolstacks can have different capabilities. If libxl is
unaware of a devices capability to provide dirty page information, but
XAPI is aware, then why is that a problem?

> I'm not 100% opposed to the idea of doing it in the toolstack; defining
> "correct" for the logdirty functionality to be "collect changes made by
> processors, I'll worry about changes made by devices" is certainly a
> reasonable approach, particularly as *somewhere* in the toolstack will
> eventually have to actually start these external emulators.
>
> But knowing that emulator X goes with bdf Y is one thing; knowing that
> 1) devices interact with logdirty, and 2) emulator X knows how to
> mitigate that is a different thing.

It is, but I don't see that Xen should have any right of veto over
what a paticular toolstack wishes to do with the domains it has
created.

  Paul

>
>  -George
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxxx
> https://lists.xenproject.org/mailman/listinfo/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel