Re: [Xen-API] XCP Storage resiliency

[George Shuklin <george.shuklin@xxxxxxxxx>]

Well, you make me curious.

Here simple test. I've done it now on XCP 1.6

1. Run fio (any kind of config, just to create some load - in my case it 
was 16 concurrent operations).
2. Put domain to pause state (xl pause) - to caught it during some IO 
'on-flight'
3. wait for 150 seconds or more
4. resume domain
amaizing, I've got some nasty lags after resume, but no IO errors. I've 
repeat that operation few times:
1. No problems.
2. Stale RCU trace: [5040301.442715] INFO: rcu_sched_state detected 
stall on CPU 0 (t=50594 jiffies)
3. No problems.

That's really strange, because I saw few times io errors on virtual 
machines without any signs of problems for the dom0.
I'll research tat topic more on next week and will report results here.


22.06.2013 02:55, Nathan March ÐÐÑÐÑ:
> On 6/21/2013 1:16 AM, George Shuklin wrote:
> > I'm talking not about dom0, mostly, but domU kernel. If IO takes more 
> > than 120 seconds, it will processed as 'io timeout'. And this timeout 
> > is hardcoded (no /sys|/proc variables).
> > If you getting IO timeout in less than 2 minutes - that different 
> > question.
> Hi George,
>
> Sorry if I'm misunderstanding, but I don't believe it's a domU issue, 
> as I've run identical virtual machines on our existing xen cluster and 
> can take storage away from the dom0 for over 45 minutes without a 
> problem. If the domU kernel was responsible for timing out the IO 
> requests I'd be seeing some sort of kernel error on my domU's in this 
> situation. Instead they just hang waiting for the IO and gracefully 
> recover once it comes back (albeit, with very very high load averages 
> as requests back up). I've done no patching/changes to our existing 
> systems to get it to work like this, it just ended up that way. We're 
> running stock 3.2.28 dom0's and 2.6.32.60 domU's, so having to hack a 
> domU kernel on XCP to achieve the same thing seems strange?
> That being said, it is a 120s timeout that I'm hitting (NFS is me 
> echoing to kmsg when I pull connectivity for easy timestamp purposes)
> dom0:
> [ 2594.069594] NFS
> [ 2609.574285] nfs: server 10.1.26.1 not responding, timed out
> [ 2717.464716] end_request: I/O error, dev tda, sector 18882056
>
> domu:
> [82688.790260] NFS
> [82812.678888] end_request: I/O error, dev xvda, sector 18882056
>
> So here the dom0 is timing out and the I/O error is returned back to 
> the domU and then it goes read only.
> If I manually unmount + remount the SR on the dom0 with "-o hard", I 
> would expect the timeout to go away as nfs is no longer returning the 
> timeout back to xcp. Instead what I see are the same 120s timeouts, 
> making me think that this timeout is coming from some other layer 
> instead?
> Thanks!
>
> - Nathan
>
>
>
> _______________________________________________
> Xen-api mailing list
> Xen-api@xxxxxxxxxxxxx
> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

_______________________________________________
Xen-api mailing list
Xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api