Re: [Xen-devel] Unstable NFS mount at heavy load.

["G.R." <firemeteor@xxxxxxxxxxxxxxxxxxxxx>]

Hi Konrad, do you have any suggestion how to debug?

Thanks,
Timothy

On Wed, Jan 9, 2013 at 4:47 PM, G.R. <firemeteor@xxxxxxxxxxxxxxxxxxxxx> wrote:
> Hi Konrad,
> Do you have any suggestion how to troubleshooting the NFS mount issue
> as described below?
> The broken connection is quite suspicious to me.
>
> Thanks,
> Timothy
>
> On Wed, Jan 9, 2013 at 1:15 AM, Stefano Stabellini
> <stefano.stabellini@xxxxxxxxxxxxx> wrote:
>> Do you mean the maintainer of the Linux PV network frontend and backend
>> drivers (netfront and netback)?
>> That would be Konrad.
>>
>> On Tue, 8 Jan 2013, G.R. wrote:
>>> Nobody responses...
>>>
>>> Stefano, could you point me to the PVNET owner?
>>> I suspect this has something to do with the net emulation.
>>>
>>> Thanks,
>>> Timothy
>>>
>>> On Sat, Jan 5, 2013 at 1:12 PM, G.R. <firemeteor@xxxxxxxxxxxxxxxxxxxxx> 
>>> wrote:
>>> > Forward this to the devel list.
>>> >
>>> >
>>> > ---------- Forwarded message ----------
>>> > From: G.R. <firemeteor@xxxxxxxxxxxxxxxxxxxxx>
>>> > Date: Sat, Jan 5, 2013 at 1:12 AM
>>> > Subject: Unstable NFS mount at heavy load.
>>> > To: xen-users@xxxxxxxxxxxxx
>>> >
>>> >
>>> > I was running benchmark on IO performance using iozone3.
>>> > In my build, the dom0 resides on a small usb stick and all the storage
>>> > comes from a NFS mount.
>>> > I test NFS performance on both dom0 && domU, mounting from the same 
>>> > server.
>>> >
>>> > The dom0 test works just well, but the domU run suffers from unstable NFS 
>>> > mount.
>>> > Since this is a NFS root, the domU just appear to be freezed.
>>> >
>>> > The log from both end of the NFS mount shows that the connection is 
>>> > broken:
>>> > Note that the client time stamp is about 20 seconds ahead of server.
>>> >
>>> > From the domU (client end):
>>> > Jan  4 23:31:16 debvm kernel: [  371.008142] nfs: server 192.168.1.8
>>> > not responding, still trying //(once)
>>> > Jan  4 23:31:25 debvm kernel: [  379.928142] nfs: server 192.168.1.8
>>> > not responding, still trying //(28 times within the same second)
>>> > Jan  4 23:31:26 debvm kernel: [  381.396143] nfs: server 192.168.1.8
>>> > not responding, still trying //(once)
>>> > Jan  4 23:31:44 debvm kernel: [  399.452129] nfs: server 192.168.1.8
>>> > not responding, still trying //(14 times within the same second)
>>> > Jan  4 23:31:45 debvm kernel: [  399.524210] nfs: server 192.168.1.8
>>> > not responding, still trying //(15 times within the same second)
>>> > Jan  4 23:31:46 debvm kernel: [  400.964142] nfs: server 192.168.1.8
>>> > not responding, still trying //(once)
>>> > Jan  4 23:31:55 debvm kernel: [  410.468787] nfs: server 192.168.1.8
>>> > OK                                     //(25 times within the same
>>> > second)
>>> > Jan  4 23:31:56 debvm kernel: [  410.520202] nfs: server 192.168.1.8
>>> > OK                                     //(32 times within the same
>>> > second)
>>> > Jan  4 23:32:05 debvm kernel: [  420.208141] nfs: server 192.168.1.8
>>> > not responding, still trying //(21 times within the same second)
>>> > Jan  4 23:32:09 debvm kernel: [  424.367613] nfs: server 192.168.1.8
>>> > OK                                     //(25 times within the same
>>> > second)
>>> > Jan  4 23:32:11 debvm kernel: [  425.764143] nfs: server 192.168.1.8
>>> > not responding, still trying
>>> > Jan  4 23:32:11 debvm kernel: [  425.772031] nfs: server 192.168.1.8 OK
>>> > Jan  4 23:32:11 debvm kernel: [  426.466328] nfs: server 192.168.1.8 OK
>>> > Jan  4 23:33:32 debvm kernel: [  507.136150] nfs: server 192.168.1.8
>>> > not responding, still trying
>>> > Jan  4 23:34:20 debvm kernel: [  555.170556] nfs: server 192.168.1.8
>>> > not responding, still trying
>>> > Jan  4 23:37:28 debvm kernel: [  742.616155] nfs: server 192.168.1.8
>>> > not responding, still trying
>>> > Jan  4 23:39:39 debvm kernel: [  873.880200] nfs: server 192.168.1.8
>>> > not responding, still trying
>>> > Jan  4 23:40:15 debvm kernel: [  909.987313] nfs: server 192.168.1.8
>>> > OK                                    //(91 times within the same
>>> > second)
>>> > Jan  4 23:40:27 debvm kernel: [  921.776152] nfs: server 192.168.1.8
>>> > not responding, still trying
>>> > Jan  4 23:40:34 debvm kernel: [  929.314639] nfs: server 192.168.1.8 OK
>>> > Jan  4 23:42:05 debvm kernel: [ 1019.584149] nfs: server 192.168.1.8
>>> > not responding, still trying
>>> > Jan  4 23:42:13 debvm kernel: [ 1028.504158] nfs: server 192.168.1.8
>>> > not responding, still trying
>>> > Jan  4 23:42:53 debvm kernel: [ 1067.565487] nfs: server 192.168.1.8
>>> > not responding, still trying
>>> > Jan  4 23:44:28 debvm kernel: [ 1163.368977] nfs: server 192.168.1.8 OK
>>> > Jan  4 23:44:33 debvm kernel: [ 1168.337859] nfs: server 192.168.1.8 OK
>>> > Jan  4 23:45:41 debvm kernel: [ 1236.448135] nfs: server 192.168.1.8
>>> > not responding, still trying
>>> > Jan  4 23:49:37 debvm kernel: [ 1471.960302] nfs: server 192.168.1.8
>>> > not responding, still trying
>>> > Jan  4 23:51:00 debvm kernel: [ 1554.982479] nfs: server 192.168.1.8 OK
>>> >
>>> > From the server side:
>>> > Jan  4 23:31:33 Hasim kernel: rpc-srv/tcp: nfsd: got error -104 when
>>> > sending 140 bytes - shutting down socket
>>> > Jan  4 23:31:33 Hasim kernel: nfsd: peername failed (err 107)!
>>> > Jan  4 23:39:50 Hasim kernel: rpc-srv/tcp: nfsd: got error -104 when
>>> > sending 140 bytes - shutting down socket
>>> > Jan  4 23:39:50 Hasim kernel: nfsd: peername failed (err 107)!
>>> > Jan  4 23:39:50 Hasim kernel: nfsd: peername failed (err 107)!
>>> > Jan  4 23:40:10 Hasim kernel: rpc-srv/tcp: nfsd: got error -104 when
>>> > sending 140 bytes - shutting down socket
>>> > Jan  4 23:44:01 Hasim kernel: rpc-srv/tcp: nfsd: got error -104 when
>>> > sending 140 bytes - shutting down socket
>>> > Jan  4 23:44:01 Hasim kernel: net_ratelimit: 11 callbacks suppressed
>>> > Jan  4 23:44:01 Hasim kernel: nfsd: peername failed (err 107)!
>>> > Jan  4 23:50:38 Hasim kernel: rpc-srv/tcp: nfsd: got error -104 when
>>> > sending 140 bytes - shutting down socket
>>> > Jan  4 23:50:38 Hasim kernel: nfsd: peername failed (err 107)!
>>> >
>>> >
>>> > Any suggestion how to debug this issue?
>>> > My xen version is 4.2.1, domU kernel is at 3.6.9, the domU is PVHVM.
>>> >
>>> > Thanks,
>>> > Timothy
>>>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel