[Xen-users] 2.6 Kernel Out of memory

["Sylvain COUTANT" <sco@xxxxxxxxxx>]

Hi guys,

 

Another strange behavior with some test servers.

 

They are booted using 2.6.11.12, xen-2.0.6, Debian.

The only daemons running on dom0 are : xend, smtp, quagga (ospf), portmap (nfs client), sshd.

 

Memory usage at startup is aroung 30 MB. Dom0 reserved memory is 256 MB.

 

The only things that happen on this server are domU's start/stop (xm create, xm shutdown) and a ssh connection every 2 seconds to get the domU's states (via xm list).

 

We know we could improve the domU states fetching but for now we are stuck to the ssh way of doing it.

 

After a few hours, the kernel begins to get out of memory and kills every process around to gather some. The only log before the server completely stops are :

 

Aug 22 22:03:37 xenu kernel: oom-killer: gfp_mask=0xd0

Aug 22 22:03:37 xenu kernel: DMA per-cpu:

Aug 22 22:03:37 xenu kernel: cpu 0 hot: low 32, high 96, batch 16

Aug 22 22:03:37 xenu kernel: cpu 0 cold: low 0, high 32, batch 16

Aug 22 22:03:37 xenu kernel: Normal per-cpu: empty

Aug 22 22:03:37 xenu kernel: HighMem per-cpu: empty

Aug 22 22:03:37 xenu kernel:

Aug 22 22:03:37 xenu kernel: Free pages:ÂÂÂÂÂÂÂ 2304kB (0kB HighMem)

Aug 22 22:03:37 xenu kernel: Active:2375 inactive:55381 dirty:0 writeback:54298 unstable:1042 free:576 slab:3961 mapped:2375 pagetables:84

Aug 22 22:03:37 xenu kernel: DMA free:2304kB min:2048kB low:2560kB high:3072kB active:9500kB inactive:221524kB present:262144kB pages_scanned:193051 all_unreclaimable? no

Aug 22 22:03:37 xenu kernel: lowmem_reserve[]: 0 0 0

Aug 22 22:03:37 xenu kernel: Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no

Aug 22 22:03:37 xenu kernel: lowmem_reserve[]: 0 0 0

Aug 22 22:03:37 xenu kernel: HighMem free:0kB min:128kB low:160kB high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no

Aug 22 22:03:37 xenu kernel: lowmem_reserve[]: 0 0 0

Aug 22 22:03:37 xenu kernel: DMA: 44*4kB 26*8kB 22*16kB 1*32kB 0*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2304kB

Aug 22 22:03:37 xenu kernel: Normal: empty

Aug 22 22:03:37 xenu kernel: HighMem: empty

Aug 22 22:03:37 xenu kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0

Aug 22 22:03:37 xenu kernel: Free swap = 0kB

Aug 22 22:03:37 xenu kernel: Total swap = 0kB

Aug 22 22:03:37 xenu kernel: Out of Memory: Killed process 1261 (exim4).

Aug 22 22:03:37 xenu kernel: oom-killer: gfp_mask=0x1d2

Aug 22 22:03:37 xenu kernel: DMA per-cpu:

Aug 22 22:03:37 xenu kernel: cpu 0 hot: low 32, high 96, batch 16

Aug 22 22:03:37 xenu kernel: cpu 0 cold: low 0, high 32, batch 16

Aug 22 22:03:37 xenu kernel: Normal per-cpu: empty

Aug 22 22:03:37 xenu kernel: HighMem per-cpu: empty

Aug 22 22:03:37 xenu kernel:

Aug 22 22:03:37 xenu kernel: Free pages:ÂÂÂÂÂÂÂ 2368kB (0kB HighMem)

Aug 22 22:03:37 xenu kernel: Active:2316 inactive:55442 dirty:0 writeback:52965 unstable:2458 free:592 slab:3936 mapped:2316 pagetables:80

Aug 22 22:03:37 xenu kernel: DMA free:2368kB min:2048kB low:2560kB high:3072kB active:9264kB inactive:221768kB present:262144kB pages_scanned:267628 all_unreclaimable? no

Aug 22 22:03:37 xenu kernel: lowmem_reserve[]: 0 0 0

Aug 22 22:03:37 xenu kernel: Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no

Aug 22 22:03:37 xenu kernel: lowmem_reserve[]: 0 0 0

Aug 22 22:03:37 xenu kernel: HighMem free:0kB min:128kB low:160kB high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no

Aug 22 22:03:37 xenu kernel: lowmem_reserve[]: 0 0 0

Aug 22 22:03:37 xenu kernel: DMA: 70*4kB 19*8kB 23*16kB 1*32kB 0*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2368kB

Aug 22 22:03:37 xenu kernel: Normal: empty

Aug 22 22:03:37 xenu kernel: HighMem: empty

Aug 22 22:03:37 xenu kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0

Aug 22 22:03:37 xenu kernel: Free swap = 0kB

Aug 22 22:03:37 xenu kernel: Total swap = 0kB

Aug 22 22:03:43 xenu kernel: Out of Memory: Killed process 1287 (ospfd).

Aug 22 22:03:43 xenu kernel: vlan0007: del 01:00:5e:00:00:06 mcast address from master interface

Aug 22 22:03:43 xenu kernel: vlan0009: del 01:00:5e:00:00:05 mcast address from master interface

Aug 22 22:03:43 xenu kernel: vlan0007: del 01:00:5e:00:00:05 mcast address from master interface

Aug 22 22:04:55 xenu kernel: oom-killer: gfp_mask=0xd0

Aug 22 22:04:55 xenu kernel: DMA per-cpu:

Aug 22 22:04:55 xenu kernel: cpu 0 hot: low 32, high 96, batch 16

Aug 22 22:04:55 xenu kernel: cpu 0 cold: low 0, high 32, batch 16

Aug 22 22:04:55 xenu kernel: Normal per-cpu: empty

Aug 22 22:04:55 xenu kernel: HighMem per-cpu: empty

Aug 22 22:04:55 xenu kernel:

Aug 22 22:04:55 xenu kernel: Free pages:ÂÂÂÂÂÂÂ 2000kB (0kB HighMem)

Aug 22 22:04:55 xenu kernel: Active:2137 inactive:55782 dirty:0 writeback:52203 unstable:3564 free:500 slab:3915 mapped:2136 pagetables:72

Aug 22 22:04:55 xenu kernel: DMA free:2000kB min:2048kB low:2560kB high:3072kB active:8548kB inactive:223128kB present:262144kB pages_scanned:223163 all_unreclaimable? no

Aug 22 22:04:55 xenu kernel: lowmem_reserve[]: 0 0 0

Aug 22 22:04:55 xenu kernel: Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no

Aug 22 22:04:55 xenu kernel: lowmem_reserve[]: 0 0 0

Aug 22 22:04:55 xenu kernel: HighMem free:0kB min:128kB low:160kB high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no

Aug 22 22:04:55 xenu kernel: lowmem_reserve[]: 0 0 0

Aug 22 22:04:55 xenu kernel: DMA: 0*4kB 24*8kB 9*16kB 6*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2000kB

Aug 22 22:04:55 xenu kernel: Normal: empty

Aug 22 22:04:55 xenu kernel: HighMem: empty

Aug 22 22:04:55 xenu kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0

Aug 22 22:04:55 xenu kernel: Free swap = 0kB

Aug 22 22:04:55 xenu kernel: Total swap = 0kB

Aug 22 22:04:55 xenu kernel: Out of Memory: Killed process 1283 (zebra).

Aug 22 22:04:55 xenu kernel: oom-killer: gfp_mask=0xd0

Aug 22 22:04:55 xenu kernel: DMA per-cpu:

Aug 22 22:04:55 xenu kernel: cpu 0 hot: low 32, high 96, batch 16

Aug 22 22:04:55 xenu kernel: cpu 0 cold: low 0, high 32, batch 16

Aug 22 22:04:55 xenu kernel: Normal per-cpu: empty

Aug 22 22:04:55 xenu kernel: HighMem per-cpu: empty

Aug 22 22:04:55 xenu kernel:

Aug 22 22:04:55 xenu kernel: Free pages:ÂÂÂÂÂÂÂ 2320kB (0kB HighMem)

Aug 22 22:04:55 xenu kernel: Active:2040 inactive:55767 dirty:0 writeback:52067 unstable:3700 free:580 slab:3915 mapped:2039 pagetables:68

Aug 22 22:04:55 xenu kernel: DMA free:2320kB min:2048kB low:2560kB high:3072kB active:8160kB inactive:223068kB present:262144kB pages_scanned:282123 all_unreclaimable? yes

Aug 22 22:04:55 xenu kernel: lowmem_reserve[]: 0 0 0

Aug 22 22:04:55 xenu kernel: Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no

Aug 22 22:04:55 xenu kernel: lowmem_reserve[]: 0 0 0

Aug 22 22:04:55 xenu kernel: HighMem free:0kB min:128kB low:160kB high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no

Aug 22 22:04:55 xenu kernel: lowmem_reserve[]: 0 0 0

Aug 22 22:04:55 xenu kernel: DMA: 34*4kB 43*8kB 11*16kB 6*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2320kB

Aug 22 22:04:55 xenu kernel: Normal: empty

Aug 22 22:04:55 xenu kernel: HighMem: empty

Aug 22 22:04:55 xenu kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0

Aug 22 22:04:55 xenu kernel: Free swap = 0kB

Aug 22 22:04:55 xenu kernel: Total swap = 0kB

Aug 22 22:04:55 xenu kernel: Out of Memory: Killed process 1142 (portmap).

Aug 22 22:04:55 xenu kernel: oom-killer: gfp_mask=0x1d2

Aug 22 22:04:55 xenu kernel: DMA per-cpu:

Aug 22 22:05:07 xenu kernel: cpu 0 hot: low 32, high 96, batch 16

Aug 22 22:05:07 xenu kernel: cpu 0 cold: low 0, high 32, batch 16

Aug 22 22:05:07 xenu kernel: Normal per-cpu: empty

Aug 22 22:05:07 xenu kernel: HighMem per-cpu: empty

Aug 22 22:05:07 xenu kernel:

Aug 22 22:05:07 xenu kernel: Free pages:ÂÂÂÂÂÂÂ 2256kB (0kB HighMem)

Aug 22 22:05:07 xenu kernel: Active:2021 inactive:55790 dirty:0 writeback:51611 unstable:4156 free:564 slab:3913 mapped:2021 pagetables:64

Aug 22 22:05:07 xenu kernel: DMA free:2256kB min:2048kB low:2560kB high:3072kB active:8084kB inactive:223160kB present:262144kB pages_scanned:153229 all_unreclaimable? no

Aug 22 22:05:07 xenu kernel: lowmem_reserve[]: 0 0 0

Aug 22 22:05:07 xenu kernel: Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no

Aug 22 22:05:07 xenu kernel: lowmem_reserve[]: 0 0 0

 

 

Could someone help us to understand how this can be xen related ? We really have never see such things on any other server before !

 

BR,

 

--

Sylvain COUTANT

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users