Re: [Xen-devel] [PATCH v4 1/2] docs/designs: Add a design document for non-cooperative live migration

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 30/01/2020 09:15, Durrant, Paul wrote:
>>> diff --git a/docs/designs/non-cooperative-migration.md
>> b/docs/designs/non-cooperative-migration.md
>>> new file mode 100644
>>> index 0000000000..5db3939db5
>>> --- /dev/null
>>> +++ b/docs/designs/non-cooperative-migration.md
>>> @@ -0,0 +1,272 @@
>>> +# Non-Cooperative Migration of Guests on Xen
>>> +
>>> +## Background
>>> +
>>> +The normal model of migration in Xen is driven by the guest because it
>> was
>>> +originally implemented for PV guests, where the guest must be aware it
>> is
>>> +running under Xen and is hence expected to co-operate.
>> For PV guests, is more than "expected to co-operate".
>>
>> Migrating a PV guest involves rewriting every pagetable entry with a
>> different MFN, so even before you consider things like the PV protocols,
>> there is no way this could be done without the cooperation of the guest.
> Yes, the P2M will change and this is visible to the guest, but does a PV 
> guest need to take action when this occurs? I'm not sure.

Consider a non-cooperative migration happening to occur between e.g.
reading a PTE and looking it up in the M2P.

I'm sure there are other cases where chaos would reign.

>> The simple fact is that involving the guest kernel adds unnecessary
>> moving parts which can (and do with a non-zero probability) go wrong.
>>
> Yes, having written the frontend side of migration in the Windows drivers it 
> is *very* hard to get right, particularly in Windows where one has to deal 
> with the complex and asynchronous PnP subsystem colliding with a migration. 
> The network driver also requires a multi-reader/single-writer lock with odd 
> semantics (w.r.t. to IRQL) which I had to code myself 
> (https://xenbits.xen.org/gitweb/?p=pvdrivers/win/xenvif.git;a=blob;f=src/xenvif/mrsw.h).
>  It took years of fixing subtle races (in that and elsewhere) to get to the 
> (AFAIK) reliable code we have now. 
> Avoiding execution of code like this (in all OS) certainly avoids the 
> opportunity for subtle bugs to manifest themselves.

There are a number of things in the guest API/ABI which are particularly
poor in retrospect.  They all need to go.

>
>>> +Because the service domain’s domid is used directly by the guest in
>> setting
>>> +up grant entries and event channels, the backend drivers in the new
>> host
>>> +environment must be provided by service domain with the same domid.
>> Also,
>>> +because the guest can sample its own domid from the frontend area and
>> use it in
>>> +hypercalls (e.g. HVMOP_set_param) rather than DOMID_SELF, the guest
>> domid must
>>> +also be preserved to maintain the ABI.
>> Has this been true since forever?  The grant and event APIs took some
>> care to avoid the guest needing to know its own domid.
>>
> The guest doesn't need to know its domid; DOMID_SELF will work, but the guest 
> *can* use its own domid in this case (whereas I think grant and event ops 
> will insist on DOMID_SELF unless referring to another domain). As far as I 
> know this has been the case since forever and so I don't think it is 
> something we can change now unless we move to a new ABI.

That smells rather like stub-qemu reasons.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel