Re: [Xen-API] sharing NFS SRs

[George Shuklin <george.shuklin@xxxxxxxxx>]

On 26.05.2012 12:57, Dave Scott wrote:
> Hi,
>
> IMHO one of the weaknesses of the current NFS SR backend in XCP is that it a 
> single SR cannot be shared between pools. This is because the backend relies on 
> the xapi pool framework to prevent
>
> 1. multiple hosts from coalescing the same vhds.
>
> 2. the same vhd being attached to two VMs at the same time.
>
> 3. a vhd being read one one node even after it has been coalesced and deleted 
> on another
>
> If multiple pools could safely share the same NFS SR then a cross-pool migrate 
> (which is possible with the current code) wouldn't have to actually mirror the 
> disks.
>
> With this in mind I've been looking into NFS locking again. I realize this is 
> a... tricky thing to get right... and google turns up lots of horror stories. 
> Anyway, here's what I was thinking:
>
> For handling (1) and (2), we would only need one lock file (really a "lease 
> file") per vhd. In the event of a network interruption we already know that running 
> VMs are likely to fail after 90s or so -- the maximum time (IIRC) a windows VM will allow 
> a page file write to take. So we could
>
> * explicitly tell tapdisk to shutdown after this long (since the VM will 
> probably have blue-screened anyway)
>
> * periodically refresh our leases, setting them to expire well after the 
> tapdisks are guaranteed to have shutdown
>
> So if a host leaves the network, all disks become unlocked a few minutes later and the 
> VMs (and coalesce jobs) can safely be restarted on another pool. This could then be used 
> as the foundation for a new "HA" feature, where only VMs whose I/Os have failed 
> are shutdown and restarted.
>
>  From an implementation point of view, this python library looks pretty good:
>
> http://bazaar.launchpad.net/~barry/flufl.lock/trunk/view/head:/flufl/lock/_lockfile.py
>
> I'm not totally sure how to handle (3): would it be sufficient to periodically 
> reopen the vhd chain in tapdisk, or just handle the error where a read fails 
> and reopen the chain then?
I've somehow afraid idea of 'leasing' operation (and periodic open/close 
operation).
Here some scenarios to think about:

1) temporal loss of the host SAN connectivity. NFS on the host is going 
to interruptible sleep and continue IO as soon as we get connectivity 
back. We already kill tapdisk, remove lease, restart vm on other host 
and suddenly networking is revived... And pending NFS write operation is 
going straight in the middle of 'mission critical' database with fresh 
'week after expiration date' data. May be weeks later after 'issue' with 
VM restart.
2) SR live migration is still very important feature I very hope to see.
3) Those leases will create additional IO. F.e. if we do have ~20k VMs 
(not really large number for clouds of new age) and lease is 10 minutes, 
it wll create ~33 IOPS - equivalent about 60-70 VMs (according to 
statistic from our cloud).
4) how do you plan to guarantee to tapdisk shutdown (this is NFS, if 
server is down or some issues with connectivity, there is no way to shut 
down locked in IO process)?
5) I think 30s is not very good number. Linux kernel starts to throwing 
IO errors after 120 seconds of IO wait.
6) about this library: '''you also need to make sure that your clocks 
are properly synchronized. """ I think this must add requirement to 
coexisting of hosts: do not allow to plug nfs sr until clock is synced 
with master. (Same for cross-pool migration - reject migration if clock 
is out of sync, but allow to shoot own leg with --force).

_______________________________________________
Xen-api mailing list
Xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api