[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-API] sharing NFS SRs

  • To: xen-api@xxxxxxxxxxxxx
  • From: George Shuklin <george.shuklin@xxxxxxxxx>
  • Date: Sat, 26 May 2012 13:50:53 +0400
  • Delivery-date: Sat, 26 May 2012 09:50:55 +0000
  • List-id: User and development list for XCP and XAPI <xen-api.lists.xen.org>

On 26.05.2012 12:57, Dave Scott wrote:

IMHO one of the weaknesses of the current NFS SR backend in XCP is that it a 
single SR cannot be shared between pools. This is because the backend relies on 
the xapi pool framework to prevent

1. multiple hosts from coalescing the same vhds.

2. the same vhd being attached to two VMs at the same time.

3. a vhd being read one one node even after it has been coalesced and deleted 
on another

If multiple pools could safely share the same NFS SR then a cross-pool migrate 
(which is possible with the current code) wouldn't have to actually mirror the 

With this in mind I've been looking into NFS locking again. I realize this is 
a... tricky thing to get right... and google turns up lots of horror stories. 
Anyway, here's what I was thinking:

For handling (1) and (2), we would only need one lock file (really a "lease 
file") per vhd. In the event of a network interruption we already know that running 
VMs are likely to fail after 90s or so -- the maximum time (IIRC) a windows VM will allow 
a page file write to take. So we could

* explicitly tell tapdisk to shutdown after this long (since the VM will 
probably have blue-screened anyway)

* periodically refresh our leases, setting them to expire well after the 
tapdisks are guaranteed to have shutdown

So if a host leaves the network, all disks become unlocked a few minutes later and the 
VMs (and coalesce jobs) can safely be restarted on another pool. This could then be used 
as the foundation for a new "HA" feature, where only VMs whose I/Os have failed 
are shutdown and restarted.

 From an implementation point of view, this python library looks pretty good:


I'm not totally sure how to handle (3): would it be sufficient to periodically 
reopen the vhd chain in tapdisk, or just handle the error where a read fails 
and reopen the chain then?

I've somehow afraid idea of 'leasing' operation (and periodic open/close operation).

Here some scenarios to think about:

1) temporal loss of the host SAN connectivity. NFS on the host is going to interruptible sleep and continue IO as soon as we get connectivity back. We already kill tapdisk, remove lease, restart vm on other host and suddenly networking is revived... And pending NFS write operation is going straight in the middle of 'mission critical' database with fresh 'week after expiration date' data. May be weeks later after 'issue' with VM restart.
2) SR live migration is still very important feature I very hope to see.
3) Those leases will create additional IO. F.e. if we do have ~20k VMs (not really large number for clouds of new age) and lease is 10 minutes, it wll create ~33 IOPS - equivalent about 60-70 VMs (according to statistic from our cloud). 4) how do you plan to guarantee to tapdisk shutdown (this is NFS, if server is down or some issues with connectivity, there is no way to shut down locked in IO process)? 5) I think 30s is not very good number. Linux kernel starts to throwing IO errors after 120 seconds of IO wait. 6) about this library: '''you also need to make sure that your clocks are properly synchronized. """ I think this must add requirement to coexisting of hosts: do not allow to plug nfs sr until clock is synced with master. (Same for cross-pool migration - reject migration if clock is out of sync, but allow to shoot own leg with --force).

Xen-api mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.