[Xen-devel] HVM network performance

[Steve Dobbelstein <steved@xxxxxxxxxx>]

I'm looking into performance issues with HVM domains.  After running some
micro benchmarks I see that network performance takes a big hit in HVM
domains.  For example, on my test machine the netperf benchmark shows that
an HVM domain gets only 3% of the throughput of a paravirtualized domain.

In an effort to track down where the time is spent I applied the patches
for Xenoprof passive domain support.  Here are the first 25 lines from
Xenoprof.

samples  %        app name                 symbol name
289967    3.6271  pxen1-syms               vmx_asm_vmexit_handler
279554    3.4968  xen-unstable-syms        hypercall
249264    3.1179  vmlinux-2.6.16-xen0-up   do_select
246991    3.0895  vmlinux-2.6.16-xen0-up   hypercall_page
225975    2.8266  vmlinux-2.6.16-xen0-up   system_call
196799    2.4617  vmlinux-2.6.16-xen0-up   schedule
150136    1.8780  pvmlinux1-syms           pcnet32_wio_write_csr
138253    1.7293  pxen1-syms               vmx_io_instruction
131584    1.6459  vmlinux-2.6.16-xen0-up   __copy_from_user_ll
128703    1.6099  pxen1-syms               vmx_vmexit_handler
111488    1.3945  vmlinux-2.6.16-xen0-up   sys_times
91488     1.1444  vmlinux-2.6.16-xen0-up   __switch_to
90813     1.1359  pvmlinux1-syms           pcnet32_wio_read_csr
90768     1.1354  libc-2.3.5.so            __GI_memcpy
86011     1.0759  vmlinux-2.6.16-xen0-up   core_sys_select
85427     1.0686  xen-unstable-syms        do_update_descriptor
79002     0.9882  vmlinux-2.6.16-xen0-up   hypervisor_callback
75150     0.9400  pxen1-syms               evtchn_set_pending
69434     0.8685  vmlinux-2.6.16-xen0-up   get_page_from_freelist
67366     0.8426  xen-unstable-syms        __copy_from_user_ll
67019     0.8383  vmlinux-2.6.16-xen0-up   __copy_to_user_ll
65826     0.8234  xen-unstable-syms        evtchn_set_pending
65719     0.8220  pxen1-syms               hvm_wait_io
65706     0.8219  pxen1-syms               get_s_time
64974     0.8127  pxen1-syms               vmx_intr_assist

The first few lines from the brief report:

  samples|      %|
------------------
  3134473 39.2076 vmlinux-2.6.16-xen0-up
  1831782 22.9129 pxen1-syms
  1472974 18.4247 xen-unstable-syms
   620624  7.7631 pvmlinux1-syms
   490539  6.1359 qemu-dm
   199750  2.4986 libc-2.3.5.so
   100124  1.2524 libpthread-2.3.5.so
    75757  0.9476 oprofiled

dom0 and the HVM domain (domain ID 1) are each running a uniprocessor
kernel.  The vcpu for the HVM domain was pinned to the sibling hyperthread
of the same core on which dom0 is running to reduce the latencies of memory
access between cores and/or sockets.

The vcpu in dom0 ran at about 83% utilized.  The vcpu in the HVM domain ran
at about 37.5% utilized.

I don't see any obvious problems in the Xenoprof output.  Anyone with
experience with the inner workings of HVM domains care to comment on what
might be causing the network performance to suffer so much?

Steve D.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel