[Xen-devel] [PATCH v9 0/3] Expose HW APIC virtualization support to HVM guests

[Boris Ostrovsky <boris.ostrovsky@xxxxxxxxxx>]

Version 9:
* Set max number of hypervisor leaves to 2 if user specifies fewer than 2.

Version 8:
* Changed minimum largest hypervisor leaf from 1 to 2 
(hvmloader.c:init_hypercalls()
  doen't like max leaf 1)
* Added 'maxhvleaf' string as alias for 0x400000XX_EAX. (No alias for Viridian
  guests though)
* Docs updates
* Changed conditions for setting XEN_HVM_CPUID_X2APIC_VIRT bit based on 
  vmx_vlapic_msr_changed() behavior. (Yang is still not convinced that 2 bits
  are needed but is OK to keep them if others agree)

(Jan, I dropped your Reviewed-by for the first patch because of the first three
bullets)

Version 7:
* Enforce user-specified number of leaves between 1 and XEN_CPUID_MAX_NUM_LEAVES
* Merge Solaris patch reversal into the first patch

Version 6:
* Drop generic support for changing hypervisor leaves, only allow modifying max
  number of leaves, ignore all other bit change requests.

Version 5:
* Remember to deal with Viridian leaves in xc_cpuid_constrain()

Version 4:
* asm changes per Jan's suggestions
* Don't pass leaf index number to handlers, check for sub_idx==0
* Added (possibly somewhat overwrought) interface to constrain user-requested
  CPUID to what is allowed (specifically, for leaf 0x40000000 only EAX[7:0]
  can be ovverwritten).

Version 3:
* Removed sysctl for querying hypervisor leaf from libxc, replaced it with
  prefixed CPUID instruction
* Limited ability change hypervisor leaves to 0x40000000
* Fixed IS_HYPERVISOR_LEAF macro

Version 2:
* Added ability to specify hypervisor CPUID leaves in config file (this requires
  new sysctl)
* Use 2 bits to indicate what is supported --- one for APIC memory access and 
the
  other for x2APIC. Still not sure whether virtual interrupt delivery should be
  exposed as well.



HVM guests running on HW that supports HW APIC virtualization features
(APIC-register virtualization, virtual interrupt delivery, etc) may
want to use APIC instead of hvm_pirqs. Since we are not guaranteed to
have these features on VMX (for example, there is a boot option to
turn it off) and there is no such support on SVM we need to make the
guest aware that its APIC accesses may not be so bad.

CPUID seems to be a good way to provide this info to the guest.

Having a guest switch to APIC shows fairly good impact on number of
VMEXITs. For example, with a pass-through NIC, netperf sees almost
half as many. Here are results for 'xentrace -e 0x00083fff -c 2 -D -T 2'

(The guest here essentially turned off XENFEAT_hvm_pirqs but we may
want to use APIC for MSI interrupts only and leave pirqs for gsi).


[root@ovs105 virt]#  cat orig  |xentrace_format  ~/xen/tools/xentrace/formats | 
awk  '{print $5}' | sort  | uniq -c
     94 cpu_change
  13944 HLT
  26341 INJ_VIRQ
  12054 INTR
  30784 INTR_WINDOW
  10126 TRAP
 124783 VMENTRY
 124782 VMEXIT
  59217 VMMCALL
     35 wrap_buffer
 
 
[root@ovs105 virt]#  cat apicv  |xentrace_format  ~/xen/tools/xentrace/formats 
| awk  '{print $5}' | sort  | uniq -c
     49 cpu_change
  16157 HLT
     31 INJ_VIRQ
  10652 INTR
     38 INTR_WINDOW
     10 NPF
  10286 TRAP
  71269 VMENTRY
  71269 VMEXIT
  34129 VMMCALL
     15 wrap_buffer

The difference is even larger when the guest is busy.

These results are in line with what has been reported for KVM. For example
http://events.linuxfoundation.org/sites/events/files/cojp13_natapov.pdf
http://www.linuxplumbersconf.org/2012/wp-content/uploads/2012/09/2012-lpc-virt-intel-vt-feat-nakajima.pdf

I am also not sure whether (cpu_has_vmx_apic_reg_virt &
cpu_has_vmx_virtualize_x2apic_mode) is sufficient to declare full HW
APIC support to a guest. The tests show ~95K VMEXITs when virtual
interrupt delivery and posted interrupts are turned off so there
appears to still be some benefit. I suppose we can use another CPUID
bit for these two (although I am not particularly eager to do this).


Boris Ostrovsky (3):
  xen/libxc: Allow changing max number of hypervisor cpuid leaves
  x86/hvm: Add HVM-specific hypervisor CPUID leaf
  x86/hvm: Indicate avaliability of HW support of APIC virtualization
    to HVM guests

 docs/man/xl.cfg.pod.5               |    8 ++++++--
 tools/libxc/xc_cpuid_x86.c          |   11 +++++++++++
 tools/libxl/libxl_cpuid.c           |    1 +
 xen/arch/x86/hvm/hvm.c              |    9 +++++++++
 xen/arch/x86/hvm/vmx/vmx.c          |   21 +++++++++++++++++++++
 xen/arch/x86/traps.c                |   20 ++++++++++++++------
 xen/include/asm-x86/hvm/hvm.h       |    7 +++++++
 xen/include/public/arch-x86/cpuid.h |   11 +++++++++++
 8 files changed, 80 insertions(+), 8 deletions(-)


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel