Xen project Mailing List

Re: [Xen-users] [Fwd: high load average on hvm gentoo linux domU]

From: Simon Bettison <simon@xxxxxxxxxxxx>

Date: Thu, 29 May 2008 16:49:25 +0100

Delivery-date: Thu, 29 May 2008 08:50:19 -0700

List-id: Xen user discussion <xen-users.lists.xensource.com>

Simon Bettison wrote: > Hi All, > > This is my first time on this (or in fact any as I recall!) list so be > gentle :) > > I am seeing some peculiar load behaviour with a gentoo linux domu which > is acting as mailserver. The load average starts out fine but then grows > over the course of several hours topping out at around 120 (one hundred > and twenty!) > > The strange thing is the machine seems responsive, imap connections > working ok, mail being delivered, snappy shell access etc > > This makes me suspicious that the load average is more a reflection of > the vm load on the cpu slice its being given (ie its only getting say > 0.01 of the cpu time, and would be running at say 0.11 if it was not > virtualised) maybe I'm way off though! > > The setup is not overly complex: > > # xm list > Name ID Mem VCPUs State > Time(s) > Domain-0 0 1708 1 r----- > 1682.4 > gentoo-spud 5 1024 1 -b---- > 2792.4 > solaris 2 1024 1 -b---- > 1757.4 > > its a gentoo-linux dom0, its running on amd-vt hardware, do both domUs > are unmodified guests... > > xen # cat /etc/xen/gentoo-spud > name = "gentoo-spud"; > memory = 1024; > disk = [ "file:/var/xen/gentoo/spud/domU,xvda,w" ]; > kernel = "/var/xen/gentoo/spud/vmlinuz"; > root = "/dev/xvda ro"; > vif = [ "mac=00:16:3E:00:00:12" ]; > > xen # cat /etc/xen/solaris.py > name = 'solaris' > memory = '1024' > disk = [ 'file:/var/xen/solaris/domU,0,w' , 'phy:/dev/sdb,1,w' , > 'phy:/dev/sdc,2,w' ,'phy:/dev/sdd,3,w' ] > vif = [ 'mac=00:16:3E:00:00:11' ] > > The solaris vm is providing zfs on the 3 physical devices ie > > > solaris # zpool list > NAME SIZE USED AVAIL CAP HEALTH ALTROOT > tank 832G 200G 632G 24% ONLINE - > > solaris # zpool status > pool: tank > state: ONLINE > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > tank ONLINE 0 0 0 > raidz1 ONLINE 0 0 0 > c0d1 ONLINE 0 0 0 > c0d2 ONLINE 0 0 0 > c0d3 ONLINE 0 0 0 > > solaris # zfs list > NAME USED AVAIL REFER MOUNTPOINT > tank 133G 412G 24.0K /tank > tank/mail 1.32G 412G 1.32G /export/zfs/mail > > the tank/mail share is exported over nfs: > > solaris # zfs get sharenfs tank/mail > NAME PROPERTY VALUE SOURCE > tank/mail sharenfs rw,anon=0 local > > the linux box is running: > postfix > courier-imapd-ssl > courier-authlib > amavisd > spamassassin > clamav > > mail is routed through postfix on port 25 to amavis on port 10024 which > takes care of spam and av then back into postfix on 10025 for local delivery > > all domains/users are virtual on postfix with delivery to > /home/vmail/<user@domain>/ > > /home/vmail is a symbolic link to /mnt/zfs/mail > /mnt/zfs/mail is nfs mounted share from the solaris vm > > through a process of elimination I found that restarting > courier-imapd-ssl would cause the loadavg to head back towards zero, > where it would stay for short while but then eventually start to > escalate again. > > I can't find any explanation.... > > # vmstat 1 > procs -----------memory---------- ---swap-- -----io---- -system-- > ----cpu---- > r b swpd free buff cache si so bi bo in cs us sy > id wa > 0 0 0 495912 14488 185348 0 0 34 64 399 296 2 1 > 96 0 > 0 0 0 494296 14488 185352 0 0 0 0 269 97 5 1 > 93 0 > 0 0 0 494296 14488 185352 0 0 0 0 207 82 0 0 > 99 0 > 0 0 0 494288 14488 185400 0 0 0 0 131 64 0 0 > 99 0 > 0 0 0 494288 14488 185452 0 0 0 0 128 60 0 0 > 100 0 > 0 0 0 494304 14488 185472 0 0 0 8 130 74 0 0 > 99 0 > 0 0 0 494304 14488 185472 0 0 0 0 144 67 0 0 > 100 0 > 0 0 0 494368 14488 185512 0 0 0 0 127 62 0 0 > 99 0 > 0 0 0 494260 14488 185512 0 0 0 0 141 73 0 0 > 100 0 > 0 0 0 494252 14488 185556 0 0 0 0 143 65 0 0 > 99 0 > 0 0 0 494252 14520 185576 0 0 0 32 116 68 0 0 > 99 0 > 0 0 0 494300 14520 185596 0 0 0 0 127 69 0 0 > 100 0 > 0 0 0 494192 14520 185596 0 0 0 0 141 68 0 0 > 99 0 > 0 0 0 494184 14520 185640 0 0 0 0 153 60 0 0 > 99 0 > 0 0 0 494184 14520 185640 0 0 0 0 122 59 0 0 > 100 0 > 0 0 0 494216 14520 185680 0 0 0 0 124 72 0 0 > 99 0 > 0 0 0 494108 14520 185680 0 0 0 0 123 67 0 0 > 100 0 > 0 0 0 494092 14520 185720 0 0 0 0 117 63 0 0 > 99 0 > 0 0 0 494092 14520 185720 0 0 0 0 119 64 0 0 > 100 0 > 0 0 0 494124 14520 185760 0 0 0 0 123 66 0 0 > 99 0 > 0 0 0 494124 14520 185760 0 0 0 0 125 70 0 0 > 99 0 > 0 0 0 494016 14520 185800 0 0 0 0 118 60 0 0 > 100 0 > 0 0 0 494016 14520 185800 0 0 0 0 93 42 0 0 > 97 0 > 0 0 0 494032 14520 185820 0 0 0 0 25 18 0 0 > 99 0 > 0 0 0 494032 14520 185820 0 0 0 0 11 12 0 0 > 100 0 > 0 0 0 494032 14520 185828 0 0 0 0 89 55 0 0 > 100 0 > 0 0 0 494032 14520 185828 0 0 0 0 138 68 0 1 > 98 0 > 0 0 0 494040 14520 185872 0 0 0 0 118 66 0 0 > 99 0 > 0 0 0 493932 14520 185872 0 0 0 0 122 63 0 0 > 99 0 > 0 0 0 493916 14520 185912 0 0 0 0 129 62 0 0 > 100 0 > 0 0 0 493916 14552 185932 0 0 0 44 126 76 0 0 > 99 0 > 0 0 0 493916 14552 185956 0 0 0 0 123 65 0 0 > 100 0 > 0 0 0 493808 14552 185956 0 0 0 0 119 58 0 0 > 99 0 > 0 0 0 493800 14552 185996 0 0 0 0 132 71 0 0 > 100 0 > 0 0 0 493800 14552 185996 0 0 0 0 120 62 0 0 > 99 0 > 0 0 0 493800 14552 186040 0 0 0 36 131 73 0 0 > 99 0 > 0 0 0 493692 14552 186040 0 0 0 0 133 68 0 0 > 100 0 > 0 0 0 493676 14552 186080 0 0 0 0 130 66 0 0 > 99 0 > 0 0 0 493676 14552 186080 0 0 0 0 125 63 0 0 > 99 0 > 0 0 0 493684 14552 186124 0 0 0 0 139 79 0 0 > 100 0 > 0 0 0 493684 14552 186124 0 0 0 0 126 68 0 0 > 99 0 > 0 0 0 493584 14552 186164 0 0 0 0 125 62 0 0 > 99 0 > 0 0 0 493584 14552 186164 0 0 0 0 127 60 0 0 > 100 0 > 0 0 0 493584 14552 186204 0 0 0 0 130 80 0 0 > 99 0 > 0 0 0 493584 14552 186204 0 0 0 0 132 69 0 0 > 99 0 > 0 0 0 493460 14564 186248 0 0 0 12 119 74 0 0 > 100 0 > 0 0 0 493460 14564 186248 0 0 0 0 129 68 0 0 > 99 0 > 0 0 0 493452 14564 186288 0 0 0 0 145 75 0 0 > 100 0 > > thought maybe it was nfs...? > > # iostat -n 1 > Linux 2.6.21-xen (spud) 05/28/08 > > Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s > wBlk_dir/s rBlk_svr/s wBlk_svr/s > swanney:/export/zfs/mail 10363.60 13.49 0.00 > 0.00 11.30 13.49 > > Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s > wBlk_dir/s rBlk_svr/s wBlk_svr/s > swanney:/export/zfs/mail 24.18 24.19 0.00 > 0.00 0.00 18.67 > > Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s > wBlk_dir/s rBlk_svr/s wBlk_svr/s > swanney:/export/zfs/mail 0.00 0.05 0.00 > 0.00 0.00 5.56 > > Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s > wBlk_dir/s rBlk_svr/s wBlk_svr/s > swanney:/export/zfs/mail 1.98 0.06 0.00 > 0.00 0.00 0.05 > > Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s > wBlk_dir/s rBlk_svr/s wBlk_svr/s > swanney:/export/zfs/mail 4137.37 0.06 0.00 > 0.00 0.00 0.07 > > Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s > wBlk_dir/s rBlk_svr/s wBlk_svr/s > swanney:/export/zfs/mail 12051.17 4.16 0.00 > 0.00 0.00 4.16 > > Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s > wBlk_dir/s rBlk_svr/s wBlk_svr/s > swanney:/export/zfs/mail 6151.21 7.25 0.00 > 0.00 0.00 7.26 > > Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s > wBlk_dir/s rBlk_svr/s wBlk_svr/s > swanney:/export/zfs/mail 15.48 15.49 0.00 > 0.00 0.00 15.49 > > Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s > wBlk_dir/s rBlk_svr/s wBlk_svr/s > swanney:/export/zfs/mail 26.00 26.00 0.00 > 0.00 0.00 26.00 > > Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s > wBlk_dir/s rBlk_svr/s wBlk_svr/s > swanney:/export/zfs/mail 26.98 27.00 0.00 > 0.00 0.00 21.00 > > Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s > wBlk_dir/s rBlk_svr/s wBlk_svr/s > swanney:/export/zfs/mail 26.79 26.79 0.00 > 0.00 0.00 29.09 > > Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s > wBlk_dir/s rBlk_svr/s wBlk_svr/s > swanney:/export/zfs/mail 20.99 21.00 0.00 > 0.00 0.00 18.77 > > Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s > wBlk_dir/s rBlk_svr/s wBlk_svr/s > swanney:/export/zfs/mail 21.20 21.24 0.00 > 0.00 0.00 27.23 > > Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s > wBlk_dir/s rBlk_svr/s wBlk_svr/s > swanney:/export/zfs/mail 0.00 0.05 0.00 > 0.00 0.00 0.05 > > Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s > wBlk_dir/s rBlk_svr/s wBlk_svr/s > swanney:/export/zfs/mail 1.98 0.05 0.00 > 0.00 0.00 0.05 > > Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s > wBlk_dir/s rBlk_svr/s wBlk_svr/s > swanney:/export/zfs/mail 0.00 0.05 0.00 > 0.00 0.00 0.05 > > Filesystem: rBlk_nor/s wBlk_nor/s rBlk_dir/s > wBlk_dir/s rBlk_svr/s wBlk_svr/s > swanney:/export/zfs/mail 2.75 2.79 0.00 > 0.00 0.00 2.79 > > the machine is hardly begin touched - only two mailboxes on it, very > litle mail passing through. > > any pointers would be really appreciated > > Simon Bettison > I'm not sure what exactly the issue was, but i think it might have something to do with courier-imap + thousands of e-mails (~20k total with some maildirs having 5k in) + NFS installed dovecot insted and this seems to be working fine. Simon _______________________________________________ Xen-users mailing list Xen-users@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-users

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.