[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] domU using linux-2.6.37-xen-next pvops kernel with CONFIG_PARAVIRT_SPINLOCKS disabled results in 150% performance improvement (updated)


  • To: Xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxx>, Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>, Jeremy Fitzhardinge <jeremy@xxxxxxxx>
  • From: Dante Cinco <dantecinco@xxxxxxxxx>
  • Date: Mon, 20 Dec 2010 17:03:13 -0800
  • Cc:
  • Delivery-date: Mon, 20 Dec 2010 17:04:49 -0800
  • Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=UUCwJkf14LCtCDXGv1F48+y3T9fW5KZL+aqesiGlXjVfza//Y2F+e7+lwZO/1j3lyU JFmX3YaqMQZiVitGNmnm8RYI/kpl3h+v9Pfvy3yjQQ8CrYYXcUUVLdwzEAi9rYi/kP6a sWCL3ijLcjaRIUsBlyrKKjC7MRXoqOavRaecg=
  • List-id: Xen developer discussion <xen-devel.lists.xensource.com>

(Sorry, I accidentally sent the previous post before finishing the summary table)

For a couple of months now, we've been trying to track down the slow I/O performance in pvops domU. Our system has 16 Fibre Channel devices, all PCI-passthrough to domU. We were previously using a 2.6.32 (Ubuntu version) HVM kernel and were getting 511k IOPS. We switched to pvops with Konrad's xen-pcifront-0.8.2 kernel and were disappointed to see the performance degrade to 11k IOPS. After disabling some kernel debug options including KMEMLEAK, the performance jumped to 186k IOPS but still well below what we were getting with the HVM kernel. We tried disabling spinlock debugging in the kernel but it actually resulted in a drop in performance to 70k IOPS.

Last week we switched to linux-2.6.37-xen-next and with the same kernel debug options disabled, the I/O performance was slightly better at 211k IOPS. We tried disabling spinlock debugging again and saw a similar drop in performance to 58k IOPS. We searched around for any performance-related posts regarding pvops and found two references to CONFIG_PARAVIRT_SPINLOCKS (one from Jeremy and one from Konrad):
http://lists.xensource.com/archives/html/xen-devel/2009-05/msg00660.html
http://lists.xensource.com/archives/html/xen-devel/2010-11/msg01111.html

Both posts recommended (Konrad strongly) enabling PARAVIRT_SPINLOCKS when running under Xen. Since it's enabled by default, we decided to see what would happen if we disabled CONFIG_PARAVIRT_SPINLOCKS. With the spinlock debugging enabled, we were getting 205k IOPS but with spinlock debugging disabled, the performance leaped to 522k IOPS !!!

I'm assuming that this behavior is unexpected.

Here's a summary of the kernels, config changes and performance (in IOPS):

                      pcifront   linux
                      0.8.2      2.6.37-xen-next
                      pvops      pvops
Spinlock
debugging enabled,     186k       205k
PARAVIRT_SPINLOCKS=y

Spinlock
debugging disabled,     70k        58k
PARAVIRT_SPINLOCKS=y

Spinlock
debugging disabled,    247k       522k
PARAVIRT_SPINLOCKS=n


system: HP Proliant DL380G7
Xen: 4.1-unstable (Thu Oct 14 12:46:29 2010 +0100 22241:3a5755249361)
dom0: 2.6.32.24-pvops-stable (xen/stable)
domU: linux-2.6.37-xen-next (2.6.37-rc5-pvops)

- Dante
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.