[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] sudden slow performance



I'm trying to track down a performance problem that has suddenly appeared on my 
Bacula server that runs as a VM. The performance hit comes when Bacula starts 
inserting values into mysql temp tables.

strace tells me that the calls go like:
write (512 bytes)
fsync()
write(1024 bytes)
fsync()
...

And each fsync call takes between 100ms and 200ms to complete, limiting the 
inserts to 5-10 per second.

All the backup IO takes place on another physical server so it's only the mysql 
throughput that is limiting me here.

iostat -x tells me that %iowait is at 100% on the DomU, and around 75% on each 
of the physical RAID1 disks on the Dom0, with throughput being 
<100kbytes/second.

I'm using Debian 3.8.x kernels from experimental, but the problem existed with 
the Debian 3.2 kernel from wheezy too.

This setup has been running fine and then suddenly started doing this.

dd with oflag=direct gives me the expected performance in both dom0 
(~90mbytes/second) and domu (~80mbytes/second), so I think the disks themselves 
are good.

smartctl says both physical drives are health and there is no other performance 
problem on the system.

I'm running out of things to point the finger at... any suggestions greatly 
appreciated!

Thanks

James

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.