|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-users] losing network packets on heavy load, xen 3.0.3
Hi Luis. It sounds very similar to our issues with Xen and bridged networking. In our case it is also a problem when NFS servers have high disk I/O (I've posted to this list about it, but no response yet). Just to verify that it indeed is a similar issue, then if you monitor the number of dropped packets on the vif in dom0, does the number of dropped packets increase when running this test? Just use 'ifconfig vifX.X' to see the status of a vif. I really like the way Xen makes us think about server management, but if these network issues isn't resolved one way or the other, then we might need to rethink our virtualization strategy. Are there other options for configuring networking in Xen than using the bridges? /me goes googling. /Daniel Portalen > From: Luis Vinay <luisvinay@xxxxxxxxx> > Date: Fri, 24 Nov 2006 13:09:08 -0300 > To: <xen-users@xxxxxxxxxxxxxxxxxxx> > Subject: [Xen-users] losing network packets on heavy load, xen 3.0.3 > > I have made this test many times with similar results all times: > > Two virtual machines, both with Debian 3.1 r3, the firs is running "iperf -s > -d" and the second runs in a infite loop "iperf -c [ip addr of machine 1] > -t 60 -N -P 10 -d" whit this creates 10(+10) parallel bidirectional > connections for 60 seconds on the server side I get this: > > vm-debian-B:~# iperf -s > ------------------------------------------------------------ > Server listening on TCP port 5001 > TCP window size: 85.3 KByte (default) > ------------------------------------------------------------ > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > [ 4] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39537 > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > ------------------------------------------------------------ > Client connecting to 10.0.41.129, TCP port 5001 > write2 failed: Broken pipe > TCP window size: 16.0 KByte (default) > ------------------------------------------------------------ > [ 6] local 0.0.0.0 port 46381 connected with 10.0.41.129 port 5001 > [ 6] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [ ... ] > [ 9] local 0.0.0.0 port 46383 connected with 10.0.41.129 port 5001 > [ 9] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [ 13] local 10.0.41.131 port 46387 connected with 10.0.41.129 port 5001 > [ 12] local 10.0.41.131 port 46386 connected with 10.0.41.129 port 5001 > [ 14] local 10.0.41.131 port 46388 connected with 10.0.41.129 port 5001 > [ 10] local 10.0.41.131 port 46384 connected with 10.0.41.129 port 5001 > [ 16] local 10.0.41.131 port 46390 connected with 10.0.41.129 port 5001 > [ 11] local 10.0.41.131 port 46385 connected with 10.0.41.129 port 5001 > [ 8] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39538 > ------------------------------------------------------------ > Client connecting to 10.0.41.129, TCP port 5001 > TCP window size: 16.0 KByte (default) > ------------------------------------------------------------ > [ 7] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39539 > connect failed: Connection refused > ------------------------------------------------------------ > Client connecting to 10.0.41.129, TCP port 5001 > TCP window size: 16.0 KByte (default) > ------------------------------------------------------------ > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > [ 32] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39540 > connect failed: Connection refused > connect failed: Connection refused > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > ------------------------------------------------------------ > Client connecting to 10.0.41.129, TCP port 5001 > TCP window size: 16.0 KByte (default) > ------------------------------------------------------------ > [ 35] local 0.0.0.0 port 46411 connected with 10.0.41.129 port 5001 > [ 35] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [ ... ] > [ 39] local 0.0.0.0 port 46415 connected with 10.0.41.129 port 5001 > [ 39] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > write2 failed: Broken pipe > [ 41] local 0.0.0.0 port 46417 connected with 10.0.41.129 port 5001 > write2 failed: Broken pipe > [ 41] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [ 40] local 0.0.0.0 port 46416 connected with 10.0.41.129 port 5001 > [ 40] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > write2 failed: Broken pipe > [ 44] local 0.0.0.0 port 46419 connected with 10.0.41.129 port 5001 > [ 44] 0.0- 0.1 sec 0.00 Bytes 0.00 bits/sec > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > write2 failed: Broken pipe > write2 failed: Broken pipe > [ 43] local 0.0.0.0 port 46418 connected with 10.0.41.129 port 5001 > [ 43] 0.0- 0.2 sec 0.00 Bytes 0.00 bits/sec > [ 34] local 0.0.0.0 port 46410 connected with 10.0.41.129 port 5001 > [ 34] 0.0- 0.2 sec 0.00 Bytes 0.00 bits/sec > [SUM] 0.0- 0.2 sec 0.00 Bytes 0.00 bits/sec > connect failed: Connection refused > [ 42] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39541 > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > write2 failed: Broken pipe > connect failed: Connection refused > write2 failed: Broken pipe > [ 30] local 0.0.0.0 port 46407 connected with 10.0.41.129 port 5001 > [ 30] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [ 31] local 0.0.0.0 port 46408 connected with 10.0.41.129 port 5001 > [ 31] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [ 22] local 10.0.41.131 port 46399 connected with 10.0.41.129 port 5001 > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Connection reset by peer > write2 failed: Broken pipe > write2 failed: Broken pipe > [ 23] local 0.0.0.0 port 46400 connected with 10.0.41.129 port 5001 > [ 23] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [ ... ] > [ 27] local 0.0.0.0 port 46404 connected with 10.0.41.129 port 5001 > [ 27] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > > write2 failed: Broken pipe > write2 failed: Connection reset by peer > write2 failed: Connection reset by peer > write2 failed: Connection reset by peer > write2 failed: Broken pipe > [ 28] local 0.0.0.0 port 46405 connected with 10.0.41.129 port 5001 > [ 28] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [ ... ] > [ 19] local 10.0.41.131 port 46396 connected with 10.0.41.129 port 5001 > [ 51] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39542 > connect failed: Connection refused > connect failed: Connection refused > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > ------------------------------------------------------------ > Client connecting to 10.0.41.129, TCP port 5001 > TCP window size: 16.0 KByte (default) > ------------------------------------------------------------ > [ 47] local 0.0.0.0 port 46422 connected with 10.0.41.129 port 5001 > [ 47] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [ ... ] > [ 48] local 0.0.0.0 port 46423 connected with 10.0.41.129 port 5001 > [ 48] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > [ 37] local 0.0.0.0 port 46428 connected with 10.0.41.129 port 5001 > [ 37] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [ ... ] > [ 35] local 0.0.0.0 port 46426 connected with 10.0.41.129 port 5001 > [ 35] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [SUM] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [ 18] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39543 > [ 33] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39544 > [ 50] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39546 > [ 61] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39545 > connect failed: Connection refused > connect failed: Connection refused > write2 failed: Connection reset by peer > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > [ 22] 0.0- 0.1 sec 11.3 KBytes 1.40 Mbits/sec > [SUM] 0.0- 0.1 sec 11.3 KBytes 1.40 Mbits/sec > write2 failed: Connection reset by peer > [ 19] 0.0- 1.2 sec 11.3 KBytes 78.0 Kbits/sec > connect failed: Connection refused > ------------------------------------------------------------ > Client connecting to 10.0.41.129, TCP port 5001 > TCP window size: 16.0 KByte (default) > ------------------------------------------------------------ > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > ------------------------------------------------------------ > Client connecting to 10.0.41.129, TCP port 5001 > TCP window size: 16.0 KByte (default) > ------------------------------------------------------------ > [ 54] local 0.0.0.0 port 46459 connected with 10.0.41.129 port 5001 > [ 54] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [ ... ] > [ 53] local 0.0.0.0 port 46458 connected with 10.0.41.129 port 5001 > [ 53] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [SUM] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > ------------------------------------------------------------ > Client connecting to 10.0.41.129, TCP port 5001 > TCP window size: 16.0 KByte (default) > ------------------------------------------------------------ > [ 65] local 0.0.0.0 port 46469 connected with 10.0.41.129 port 5001 > [ 65] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [ ... ] > [ 64] local 0.0.0.0 port 46468 connected with 10.0.41.129 port 5001 > [ 64] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [SUM] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > ------------------------------------------------------------ > Client connecting to 10.0.41.129, TCP port 5001 > TCP window size: 16.0 KByte (default) > ------------------------------------------------------------ > [ 75] local 0.0.0.0 port 46479 connected with 10.0.41.129 port 5001 > [ 75] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [ ... ] > [ 67] local 0.0.0.0 port 46471 connected with 10.0.41.129 port 5001 > [ 67] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [SUM] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > [ 68] local 0.0.0.0 port 46472 connected with 10.0.41.129 port 5001 > [ 68] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [ ... ] > [ 74] local 0.0.0.0 port 46478 connected with 10.0.41.129 port 5001 > [ 74] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > ------------------------------------------------------------ > Client connecting to 10.0.41.129, TCP port 5001 > TCP window size: 16.0 KByte (default) > ------------------------------------------------------------ > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > write2 failed: Broken pipe > connect failed: Connection refused > connect failed: Connection refused > connect failed: Connection refused > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > [ 17] local 0.0.0.0 port 46438 connected with 10.0.41.129 port 5001 > [ 17] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [ ... ] > [SUM] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > write2 failed: Broken pipe > [ 40] local 0.0.0.0 port 46432 connected with 10.0.41.129 port 5001 > [ 40] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > [ ... ] > [ 23] local 0.0.0.0 port 46437 connected with 10.0.41.129 port 5001 > [ 23] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > write2 failed: Broken pipe > write2 failed: Broken pipe > [ 20] local 0.0.0.0 port 46439 connected with 10.0.41.129 port 5001 > [ ... ] > [ 5] 0.0-57.9 sec 11.1 MBytes 1.61 Mbits/sec > > the client side shows this: > Debian31r1-vm1:~# iperf -c 10.0.41.131 -t 60 -N -P 10 -d > ------------------------------------------------------------ > Server listening on TCP port 5001 > TCP window size: 85.3 KByte (default) > ------------------------------------------------------------ > ------------------------------------------------------------ > Client connecting to 10.0.41.131, TCP port 5001 > TCP window size: 16.0 KByte (default) > ------------------------------------------------------------ > [ 4] local 10.0.41.129 port 39538 connected with 10.0.41.131 port 5001 > [ 6] local 10.0.41.129 port 39540 connected with 10.0.41.131 port 5001 > [ 8] local 10.0.41.129 port 39542 connected with 10.0.41.131 port 5001 > [ 7] local 10.0.41.129 port 39541 connected with 10.0.41.131 port 5001 > [ 9] local 10.0.41.129 port 39543 connected with 10.0.41.131 port 5001 > [ 10] local 10.0.41.129 port 39544 connected with 10.0.41.131 port 5001 > [ 3] local 10.0.41.129 port 39537 connected with 10.0.41.131 port 5001 > [ 13] local 10.0.41.129 port 39546 connected with 10.0.41.131 port 5001 > [ 5] local 10.0.41.129 port 39539 connected with 10.0.41.131 port 5001 > [ 14] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46384 > [ 15] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46385 > [ 16] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46386 > [ 17] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46387 > [ 18] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46388 > [ 19] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46389 > [ 11] local 10.0.41.129 port 39545 connected with 10.0.41.131 port 5001 > [ 20] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46391 > [ 22] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46392 > [ 23] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46393 > [ 21] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46390 > [ 4] 0.0-60.0 sec 11.6 MBytes 1.62 Mbits/sec > [ 7] 0.0-60.0 sec 11.8 MBytes 1.65 Mbits/sec > [ 6] 0.0-60.0 sec 12.2 MBytes 1.71 Mbits/sec > [ 14] 0.0-60.0 sec 18.1 MBytes 2.53 Mbits/sec > [ 16] 0.0-60.0 sec 14.1 MBytes 1.97 Mbits/sec > [ 18] 0.0-60.0 sec 14.3 MBytes 2.00 Mbits/sec > [ 19] 0.0-60.0 sec 12.5 MBytes 1.75 Mbits/sec > [ 21] 0.0-60.0 sec 13.2 MBytes 1.85 Mbits/sec > [ 15] 0.0-60.0 sec 14.4 MBytes 2.01 Mbits/sec > [ 13] 0.0-60.3 sec 10.1 MBytes 1.40 Mbits/sec > [ 9] 0.0-60.3 sec 8.66 MBytes 1.20 Mbits/sec > [ 17] 0.0-60.2 sec 12.2 MBytes 1.71 Mbits/sec > [ 8] 0.0-60.5 sec 4.96 MBytes 688 Kbits/sec > [ 11] 0.0-60.5 sec 11.0 MBytes 1.53 Mbits/sec > [ 23] 0.0-60.6 sec 17.6 MBytes 2.44 Mbits/sec > [ 5] 0.0-60.7 sec 12.8 MBytes 1.77 Mbits/sec > [ 22] 0.0-60.6 sec 14.0 MBytes 1.93 Mbits/sec > [ 3] 0.0-60.7 sec 12.8 MBytes 1.77 Mbits/sec > [ 10] 0.0-61.1 sec 9.49 MBytes 1.30 Mbits/sec > [SUM] 0.0-61.1 sec 105 MBytes 14.5 Mbits/sec > [ 20] 0.0-61.7 sec 11.1 MBytes 1.52 Mbits/sec > [SUM] 0.0-61.7 sec 142 MBytes 19.3 Mbits/sec > > As we can see there is a lot of connection errors and the speed measurement > reveals a very poor performance > > Test platform detail: > > Dom0 > Dell PE1950, 4 Xeon @1.6G, 8Gig memory > RedHat Enterprise Server 4 update 4 > Xen 3.0.3, kernel 2.6.16.29 > > Virtual Machines > 1 CPU, 300Mb memory > Debian 3.1r3 > Kernel 2.6.8-3-386 > > Both machines are on the same Dom0 and the use bridged networking > > Any ideas or comments? > _______________________________________________ > Xen-users mailing list > Xen-users@xxxxxxxxxxxxxxxxxxx > http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-users
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |