[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-API] [XCP-1.1] High OVS cpu load and unresponsive host network while VMPR archive phase is running

  • To: xen-api@xxxxxxxxxxxxx
  • From: George Shuklin <george.shuklin@xxxxxxxxx>
  • Date: Fri, 03 Aug 2012 01:46:18 +0400
  • Delivery-date: Thu, 02 Aug 2012 21:46:29 +0000
  • List-id: User and development list for XCP and XAPI <xen-api.lists.xen.org>

In product environment I saw that behavior few times. ovs-* processes starts to consume lot of cpu (over 100%) and start to cause packets drops.

That usually happens with 'hacked' customer VMs (sudden spike of outgoing traffic, cpu, and in few cases we assisted in research, actual trojans running on server because of some stupid php misconfiguration in yet another phpbb/cms/durpal/etc).

I'm not sure wat exactly happens, but my hypothesis is that it related to amount of flows. Then trojan starts to flood out traffic to different servers (smtp/www spam/etc) it cause lots of new connections..

On 01.08.2012 04:08, Ben Pfaff wrote:
Christian Fischer<christian.fischer@xxxxxxxxxxxxxxxxxxx>  writes:

On Tuesday 31 July 2012 18:08:18 Ben Pfaff wrote:
Christian Fischer

We have no tagged vlans here, all physical switch ports running access
mode. I wouldn't say that network load is increased when this happens,
15 kpps. Network performance could be poor due either a vswitch issue
(runs at 180% CPU load if the vswitch log don't lie) or high load
on/cheep hardware of the customer shared backup storage. I've never seen
this stuff.
180% CPU load is impossible for OVS 1.0.1, which has only a
single procsss with a single thread.
Yes, that's right, but we run OVS 1.4.2

XCP build: 1.1.0-50674c
OVS build: 1.4.2
NICs: BCM5709 Gigabit TOE iSCSI Offload
OVS NIC bonding: active/active
Only the as-yet-unreleased post-1.8.0 Open vSwitch has more than
one process, and it still doesn't have multiple threads.

I suppose ovsdb-server and ovs-vswitchd could both go crazy at
the same time, but I haven't had any reports of that.

What process(es) add up to 180%?

Xen-api mailing list

Xen-api mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.