Re: [Xen-devel] [PATCH] Network Checksum Removal

[Andrew Theurer <habanero@xxxxxxxxxx>]

Tests for domU->dom0, domU->host, and domU->domU are completed:

3.2 GHz Xeon with Hyperhtreading, 2GB (correction) memory
                                                                                
                                                    
Benchmark: netperf2 -T TCP_STREAM
                                                                                
                                                    
dom0, dom1, and dom2 on cpu0 (first SMT thread on first core)
 domU to host
  hw tx csum
   msg-size: 00064  Mbps: 0186  d0-cpu: 49.38  d1-cpu: 44.35
   msg-size: 01500  Mbps: 0917  d0-cpu: 62.13  d1-cpu: 37.87
   msg-size: 16384  Mbps: 0933  d0-cpu: 66.63  d1-cpu: 33.37
   msg-size: 32768  Mbps: 0928  d0-cpu: 66.96  d1-cpu: 32.66
  sw tx csum
   msg-size: 00064  Mbps: 0187  d0-cpu: 49.50  d1-cpu: 44.52
   msg-size: 01500  Mbps: 0904  d0-cpu: 60.63  d1-cpu: 39.36
   msg-size: 16384  Mbps: 0924  d0-cpu: 63.98  d1-cpu: 35.98
   msg-size: 32768  Mbps: 0926  d0-cpu: 64.18  d1-cpu: 35.68
        ^^about 2% reduction in cpu util on dom1^^
 domU to dom0
  hw tx csum
   msg-size: 00064  Mbps: 0014  d0-cpu: 64.02  d1-cpu: 31.71
   msg-size: 01500  Mbps: 1087  d0-cpu: 63.34  d1-cpu: 36.67
   msg-size: 16384  Mbps: 1204  d0-cpu: 67.30  d1-cpu: 32.71
   msg-size: 32768  Mbps: 1148  d0-cpu: 68.08  d1-cpu: 31.93
  sw tx csum
   msg-size: 00064  Mbps: 0014  d0-cpu: 64.88  d1-cpu: 32.39
   msg-size: 01500  Mbps: 0948  d0-cpu: 62.20  d1-cpu: 37.80
   msg-size: 16384  Mbps: 1063  d0-cpu: 64.73  d1-cpu: 35.27
   msg-size: 32768  Mbps: 1012  d0-cpu: 65.71  d1-cpu: 34.30
        ^^upto 13% throughput increase with cpu util down ~2% on dom1^^
          Note the dismal performance for very small msg sizes
 donU to domU
  hw tx csum
   msg-size:00064 Mbps: 0359  d0-cpu: 27.85  d1-cpu: 53.68 d2-cpu: 18.48
   msg-size:01500 Mbps: 0594  d0-cpu: 47.42  d1-cpu: 21.77 d2-cpu: 30.78
   msg-size:16384 Mbps: 0619  d0-cpu: 49.66  d1-cpu: 18.81 d2-cpu: 31.53
   msg-size:32768 Mbps: 0616  d0-cpu: 49.58  d1-cpu: 18.68 d2-cpu: 31.74
  sw tx csum
   msg-size:00064 Mbps: 0361  d0-cpu: 27.81  d1-cpu: 53.58 d2-cpu: 18.62
   msg-size:01500 Mbps: 0584  d0-cpu: 46.22  d1-cpu: 23.18 d2-cpu: 30.60
   msg-size:16384 Mbps: 0602  d0-cpu: 47.99  d1-cpu: 20.33 d2-cpu: 31.69
   msg-size:32768 Mbps: 0603  d0-cpu: 47.67  d1-cpu: 20.59 d2-cpu: 31.74
        ^^About a 2% throughput increase, and cpu down on d1
          The cpu wasted on dom1 should be enough justification for
          domU<->domU communication with point to point front end driver
          communication.  
dom0 on cpu0, dom1 on cpu2, and dom2 on cpu3 (dom1 and dom2 on same 
core)
 domU to host
  hw tx csum
   msg-size: 00064  Mbps: 0540  d0-cpu: 92.98  d1-cpu: 100.00
   msg-size: 01500  Mbps: 0941  d0-cpu: 99.74  d1-cpu: 48.62
   msg-size: 16384  Mbps: 0941  d0-cpu: 99.71  d1-cpu: 43.32
   msg-size: 32768  Mbps: 0941  d0-cpu: 99.72  d1-cpu: 43.21
  sw tx csum
   msg-size: 00064  Mbps: 0545  d0-cpu: 93.47  d1-cpu: 100.00
   msg-size: 01500  Mbps: 0941  d0-cpu: 99.76  d1-cpu: 51.43
   msg-size: 16384  Mbps: 0941  d0-cpu: 99.69  d1-cpu: 46.58
   msg-size: 32768  Mbps: 0941  d0-cpu: 99.72  d1-cpu: 45.39
        ^^Finally at wire speed, but at a cost of 100% cpu on dom0
          This cpu util seems excessive, maybe oprofile will show
          some problems.  Notice dom1 has ~2% lower cpu.
 domU to dom0
  tx csum
   msg-size: 00064  Mbps: 0390  d0-cpu: 97.92  d1-cpu: 100.00
   msg-size: 01500  Mbps: 1571  d0-cpu: 97.36  d1-cpu: 54.83
   msg-size: 16384  Mbps: 1582  d0-cpu: 96.20  d1-cpu: 49.93
   msg-size: 32768  Mbps: 1596  d0-cpu: 96.32  d1-cpu: 49.63
  sw tx csum
   msg-size: 00064  Mbps: 0375  d0-cpu: 97.65  d1-cpu: 100.00 
   msg-size: 01500  Mbps: 1546  d0-cpu: 96.36  d1-cpu: 52.99
   msg-size: 16384  Mbps: 1598  d0-cpu: 95.88  d1-cpu: 47.48
   msg-size: 32768  Mbps: 1641  d0-cpu: 95.89  d1-cpu: 46.37 
        ^^very slightly better avg throughput, and lower cpu on dom1
 donU to domU
  tx csum
   msg-size:00064 Mbps: 0287  d0-cpu: 84.97  d1-cpu: 100.0 d2-cpu: 75.46
   msg-size:01500 Mbps: 1004  d0-cpu: 90.98  d1-cpu: 68.29 d2-cpu: 76.94
   msg-size:16384 Mbps: 1018  d0-cpu: 89.78  d1-cpu: 60.82 d2-cpu: 78.12
   msg-size:32768 Mbps: 1010  d0-cpu: 89.30  d1-cpu: 59.83 d2-cpu: 77.99
  sw tx csum
   msg-size:00064 Mbps: 0286  d0-cpu: 84.81  d1-cpu: 99.93 d2-cpu: 76.28
   msg-size:01500 Mbps: 1018  d0-cpu: 91.30  d1-cpu: 67.27 d2-cpu: 75.08
   msg-size:16384 Mbps: 1012  d0-cpu: 88.46  d1-cpu: 55.56 d2-cpu: 71.37
   msg-size:32768 Mbps: 1017  d0-cpu: 88.33  d1-cpu: 54.96 d2-cpu: 70.96
        ^^about same throughput, but ~4% lower cpu on d1
          Again, point to point front end comms woudl be great here.


IMO, I think the patch is a good thing.  There are other very major 
issues with networking, like the massive cpu overhead for dom0.  I 
wonder if we could have a layer 2 networking model like:

-Xen has have front end ethernet drivers only
-dom0 has a Xen bridge front end driver, just to put eth0 (or whatever 
phys dev) on it.
-no domain hosted bridge device or backend ethernet drivers
 
With this, Xen acts as a ethernet "switch", switching ethernet traffic 
in xen itself, without the help of a domain hosted bridge.  Packets are 
forwarded to either a domain's front end driver, or the front end 
bridge interface in dom0 (or any other driver domain).  With this we 
may have better control of emulating offload functions, and we should 
avoid some hops (and in may cases involving dom0) for the netwrok 
traffic.  Comments?

-Andrew                                                                         
                                                       



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel