[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [Xen-users] unexpected Out Of Memory (OOM)



On Wed, 2013-08-07 at 02:02 +0200, Olivier Bonvalet wrote:
> Hi,
> 
> I have a memory problem on several PV domU, I *suppose* since the
> upgrade to 3.10.4 kernel (and still occur with 3.10.5).
> I use Xen 4.2.2 (from Debian sid), and the dom0 run under Debian Wheezy
> with a vanilla 3.9.11 kernel.
> 
> The problem : once a domU throw oom-killer, then it loop until all
> processes will be killed, then the kernel abort and shutdown the VM.
> 
> I'm not sure how to read the OOM-killer messages about memory state, so
> I copy/paste a list of them here, about a 2GB RAM domU :

Does /proc/meminfo on the VM show that it actually has 2GB?

> [56313.921381] cron invoked oom-killer: gfp_mask=0x280da, order=0, 
> oom_score_adj=0
> [56313.921407] CPU: 0 PID: 7001 Comm: cron Not tainted 2.6.50-dae-xen #2

This appears to show a 2.6.50 kernel, whatever that is, and not 3.10.x?

> [56313.921413]  0000000000000000 ffffffff8134ab12 ffff880079cd2120 
> 0000000000000000
> [56313.921424]  00000000000280da ffffffff81089ad6 ffffffff8134f09a 
> ffff88007773bba0
> [56313.921433]  ffffffff814d3d48 0000000000000000 00000000000280da 
> 000000000000000b
> [56313.921443] Call Trace:
> [56313.921454]  [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> [56313.921462]  [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> [56313.921470]  [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> [56313.921480]  [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> [56313.921486]  [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> [56313.921493]  [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> [56313.921502]  [<ffffffff810da1db>] ? alloc_pages_vma+0xd4/0xfc
> [56313.921510]  [<ffffffff810cf31f>] ? anon_vma_alloc+0x11/0x21
> [56313.921516]  [<ffffffff810c7ff1>] ? handle_pte_fault+0x60f/0x6ed
> [56313.921538]  [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> [56313.921547]  [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> [56313.921560]  [<ffffffff81072f4b>] ? lg_local_lock+0x11/0x14
> [56313.921566]  [<ffffffff810fee0e>] ? mntput_no_expire+0x2d/0x133
> [56313.921572]  [<ffffffff81072f5f>] ? lg_local_unlock+0x11/0x14
> [56313.921579]  [<ffffffff810f3dca>] ? do_last+0x8eb/0x9c3
> [56313.921587]  [<ffffffff811b0401>] ? string.isra.5+0x37/0x9b
> [56313.921593]  [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> [56313.921600]  [<ffffffff81101c75>] ? seq_read+0xb8/0x31a
> [56313.921606]  [<ffffffff811b185d>] ? copy_user_generic_string+0x2d/0x40
> [56313.921612]  [<ffffffff81101e60>] ? seq_read+0x2a3/0x31a
> [56313.921632]  [<ffffffff81129c95>] ? proc_reg_read+0x54/0x74
> [56313.921644]  [<ffffffff810e977f>] ? vfs_read+0x8e/0xf0
> [56313.921650]  [<ffffffff810e9e38>] ? SyS_read+0x41/0x70
> [56313.921656]  [<ffffffff8134fc69>] ? system_call_fastpath+0x16/0x1b
> [56313.921662] Mem-Info:
> [56313.921671] Node 0 DMA per-cpu:
> [56313.921676] CPU    0: hi:    0, btch:   1 usd:   0
> [56313.921680] Node 0 DMA32 per-cpu:
> [56313.921684] CPU    0: hi:  186, btch:  31 usd:  83
> [56313.921693] active_anon:22546 inactive_anon:1033 isolated_anon:1
> [56313.921693]  active_file:153 inactive_file:849 isolated_file:0
> [56313.921693]  unevictable:0 dirty:0 writeback:0 unstable:0
> [56313.921693]  free:3392 slab_reclaimable:101577 slab_unreclaimable:232545
> [56313.921693]  mapped:3613 shmem:8327 pagetables:2848 bounce:0
> [56313.921693]  free_cma:0
> [56313.921708] Node 0 DMA free:7936kB min:44kB low:52kB high:64kB 
> active_anon:132kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB 
> managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB 
> slab_reclaimable:1568kB slab_unreclaimable:3820kB kernel_stack:2056kB 
> pagetables:360kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB 
> pages_scanned:0 all_unreclaimable? yes
> [56313.921726] lowmem_reserve[]: 0 1973 1973 1973
> [56313.921732] Node 0 DMA32 free:5632kB min:5660kB low:7072kB high:8488kB 
> active_anon:90052kB inactive_anon:4132kB active_file:612kB 
> inactive_file:3396kB unevictable:0kB isolated(anon):4kB isolated(file):0kB 
> present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB 
> mapped:14452kB shmem:33308kB slab_reclaimable:404740kB 
> slab_unreclaimable:926360kB kernel_stack:572928kB pagetables:11032kB 
> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2541 
> all_unreclaimable? yes
> [56313.921749] lowmem_reserve[]: 0 0 0 0
> [56313.921757] Node 0 DMA: 56*4kB (UE) 20*8kB (UE) 14*16kB (UE) 21*32kB (UE) 
> 22*64kB (UE) 9*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 
> 7936kB
> [56313.921775] Node 0 DMA32: 204*4kB (EM) 492*8kB (EM) 15*16kB (UMR) 0*32kB 
> 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5632kB
> [56313.921791] 9341 total pagecache pages
> [56313.921794] 0 pages in swap cache
> [56313.921797] Swap cache stats: add 0, delete 0, find 0/0
> [56313.921799] Free swap  = 0kB
> [56313.921802] Total swap = 0kB
> [56313.927968] 524287 pages RAM
> [56313.927976] 12090 pages reserved
> [56313.927978] 307132 pages shared
> [56313.927980] 430905 pages non-shared
> [56313.927983] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents 
> oom_score_adj name
> [56313.927996] [  330]     0   330     4213      114      12        0         
> -1000 udevd
> [56313.928001] [  815]     0   815    13018      114      20        0         
>     0 rsyslogd
> [56313.928006] [  853]   104   853    12487      102      27        0         
>     0 zabbix_agentd
> [56313.928011] [  909]     0   909     4212      113      12        0         
> -1000 udevd
> [56313.928015] [  910]     0   910     4212      113      12        0         
> -1000 udevd
> [56313.928020] [  911]   104   911    12487      207      27        0         
>     0 zabbix_agentd
> [56313.928025] [  912]   104   912    12487      101      27        0         
>     0 zabbix_agentd
> [56313.928029] [  913]   104   913    12487      101      27        0         
>     0 zabbix_agentd
> [56313.928034] [  914]   104   914    12487      101      27        0         
>     0 zabbix_agentd
> [56313.928038] [  915]   104   915    13005      109      29        0         
>     0 zabbix_agentd
> [56313.928043] [ 1016]     0  1016    86937     2033     155        0         
>     0 apache2
> [56313.928048] [ 1026]     0  1026     4692       40      14        0         
>     0 atd
> [56313.928057] [ 1054]     0  1054     5747       69      16        0         
>     0 cron
> [56313.928062] [ 1076]   103  1076     5830       54      15        0         
>     0 dbus-daemon
> [56313.928067] [ 1104]   106  1104     9630      147      23        0         
>     0 ntpd
> [56313.928072] [ 1117]     8  1117     2947       45      12        0         
>     0 nullmailer-send
> [56313.928077] [ 1130]     0  1130    18989      374      35        0         
>     0 searchd
> [56313.928081] [ 1140]     0  1140    12307      138      28        0         
> -1000 sshd
> [56313.928086] [ 1152]   105  1152    10265     1088      24        0         
>     0 unbound
> [56313.928090] [ 1172]     0  1172     1496       25       9        0         
>     0 getty
> [56313.928150] [ 1173]     0  1173     1496       24       8        0         
>     0 getty
> [56313.928157] [31557]    33 31557    88162     5277     161        0         
>     0 apache2
> [56313.928163] [31998]    33 31998    88044     5689     162        0         
>     0 apache2
> [56313.928168] [ 1031]    33  1031    87879     4726     161        0         
>     0 apache2
> [56313.928172] [ 1579]    33  1579    87493     5027     158        0         
>     0 apache2
> [56313.928177] [ 2901]    33  2901    87622     4707     158        0         
>     0 apache2
> [56313.928182] [ 3447]    33  3447    82447     4550     150        0         
>     0 apache2
> [56313.928186] [ 4559]    33  4559    87714     4945     160        0         
>     0 apache2
> [56313.928191] [ 4560]    33  4560    82914     4906     150        0         
>     0 apache2
> [56313.928195] [ 4561]    33  4561    87859     3840     159        0         
>     0 apache2
> [56313.928200] [ 6205]    33  6205    87422     3053     154        0         
>     0 apache2
> [56313.928204] [ 6776]     0  6776    10460       89      25        0         
>     0 cron
> [56313.928229] [ 6780]     0  6780     1002       23       8        0         
>     0 sh
> [56313.928236] [ 6784]     0  6784     7322      109      20        0         
>     0 wget
> [56313.928241] [ 6847]     0  6847    10460       89      25        0         
>     0 cron
> [56313.928246] [ 6856]     0  6856     1002       23       8        0         
>     0 sh
> [56313.928253] [ 6862]     0  6862     7322      108      20        0         
>     0 wget
> [56313.928258] [ 6924]     0  6924    10460       89      25        0         
>     0 cron
> [56313.928263] [ 6928]     0  6928     1002       24       8        0         
>     0 sh
> [56313.928267] [ 6932]     0  6932     7322      107      20        0         
>     0 wget
> [56313.928272] [ 6948]    33  6948    82020     2026     135        0         
>     0 apache2
> [56313.928277] [ 6991]    33  6991    86937     1966     137        0         
>     0 apache2
> [56313.928281] [ 6992]    33  6992    86937     1966     137        0         
>     0 apache2
> [56313.928285] [ 6993]     0  6993    10460      106      25        0         
>     0 cron
> [56313.928290] [ 6994]     0  6994    10460      106      25        0         
>     0 cron
> [56313.928294] [ 6995]     0  6995    10460      106      25        0         
>     0 cron
> [56313.928298] [ 6999]  1337  6999    10459      183      23        0         
>     0 cron
> [56313.928302] [ 7000]     0  7000    10459      182      23        0         
>     0 cron
> [56313.928306] [ 7001]     0  7001    10460      182      23        0         
>     0 cron
> [56313.928311] Out of memory: Kill process 31998 (apache2) score 11 or 
> sacrifice child
> [56313.928315] Killed process 31998 (apache2) total-vm:352176kB, 
> anon-rss:10848kB, file-rss:11908kB
> [56317.522091] searchd invoked oom-killer: gfp_mask=0x2840d0, order=0, 
> oom_score_adj=0
> [56317.522109] CPU: 0 PID: 1130 Comm: searchd Not tainted 2.6.50-dae-xen #2
> [56317.522115]  0000000000000000 ffffffff8134ab12 0000000000000080 
> ffff8800793078e8
> [56317.522125]  0000000000000000 ffffffff81089ad6 ffffffff8134f09a 
> ffff880001802120
> [56317.522133]  ffffffff814d3d48 0000000000000000 00000000002840d0 
> 000000000000000a
> [56317.522142] Call Trace:
> [56317.522153]  [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> [56317.522161]  [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> [56317.522168]  [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> [56317.522177]  [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> [56317.522184]  [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> [56317.522190]  [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> [56317.522197]  [<ffffffff810deff1>] ? new_slab+0x6e/0x1d5
> [56317.522203]  [<ffffffff8134b905>] ? __slab_alloc+0x28f/0x38b
> [56317.522211]  [<ffffffff811ac4bc>] ? radix_tree_preload+0x30/0x77
> [56317.522219]  [<ffffffff81111258>] ? mpage_alloc+0x25/0x7d
> [56317.522226]  [<ffffffff81140128>] ? _ext4_get_block+0x141/0x141
> [56317.522232]  [<ffffffff810e00b4>] ? kmem_cache_alloc+0x3d/0x8f
> [56317.522238]  [<ffffffff811ac4bc>] ? radix_tree_preload+0x30/0x77
> [56317.522245]  [<ffffffff810ae675>] ? add_to_page_cache_locked+0x1b/0x92
> [56317.522251]  [<ffffffff810ae6fa>] ? add_to_page_cache_lru+0xe/0x31
> [56317.522258]  [<ffffffff8111185f>] ? mpage_readpages+0x9c/0x105
> [56317.522264]  [<ffffffff81140128>] ? _ext4_get_block+0x141/0x141
> [56317.522271]  [<ffffffff810b6aeb>] ? __do_page_cache_readahead+0x112/0x1ab
> [56317.522277]  [<ffffffff810b6ddb>] ? ra_submit+0x18/0x1b
> [56317.522283]  [<ffffffff810b0309>] ? filemap_fault+0x173/0x35f
> [56317.522289]  [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> [56317.522296]  [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> [56317.522304]  [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> [56317.522310]  [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> [56317.522440]  [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> [56317.522449]  [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> [56317.522457]  [<ffffffff81113891>] ? fsnotify+0x261/0x291
> [56317.522464]  [<ffffffff81350efb>] ? xen_hypervisor_callback+0x1b/0x20
> [56317.522471]  [<ffffffff81005319>] ? xen_clocksource_read+0x1c/0x1d
> [56317.522479]  [<ffffffff8107f3fc>] ? 
> timekeeping_get_ns.constprop.10+0xd/0x31
> [56317.522498]  [<ffffffff8107f778>] ? ktime_get_ts+0x35/0x72
> [56317.522503]  [<ffffffff810f6d92>] ? poll_select_copy_remaining+0xde/0xf3
> [56317.522508]  [<ffffffff81005319>] ? xen_clocksource_read+0x1c/0x1d
> [56317.522512]  [<ffffffff8107f3fc>] ? 
> timekeeping_get_ns.constprop.10+0xd/0x31
> [56317.522517]  [<ffffffff8107f8e7>] ? __getnstimeofday+0x1b/0x55
> [56317.522522]  [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> [56317.522526] Mem-Info:
> [56317.522529] Node 0 DMA per-cpu:
> [56317.522540] CPU    0: hi:    0, btch:   1 usd:   0
> [56317.522544] Node 0 DMA32 per-cpu:
> [56317.522549] CPU    0: hi:  186, btch:  31 usd:  90
> [56317.522556] active_anon:22815 inactive_anon:1033 isolated_anon:0
> [56317.522556]  active_file:95 inactive_file:793 isolated_file:0
> [56317.522556]  unevictable:0 dirty:0 writeback:0 unstable:0
> [56317.522556]  free:3378 slab_reclaimable:101712 slab_unreclaimable:232523
> [56317.522556]  mapped:3140 shmem:8327 pagetables:2478 bounce:0
> [56317.522556]  free_cma:0
> [56317.522572] Node 0 DMA free:7936kB min:44kB low:52kB high:64kB 
> active_anon:132kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB 
> managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB 
> slab_reclaimable:1568kB slab_unreclaimable:3820kB kernel_stack:2056kB 
> pagetables:360kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB 
> pages_scanned:0 all_unreclaimable? yes
> [56317.522590] lowmem_reserve[]: 0 1973 1973 1973
> [56317.522599] Node 0 DMA32 free:5576kB min:5660kB low:7072kB high:8488kB 
> active_anon:91128kB inactive_anon:4132kB active_file:380kB 
> inactive_file:3172kB unevictable:0kB isolated(anon):0kB isolated(file):0kB 
> present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB 
> mapped:12560kB shmem:33308kB slab_reclaimable:405280kB 
> slab_unreclaimable:926272kB kernel_stack:572976kB pagetables:9552kB 
> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3288 
> all_unreclaimable? yes
> [56317.522621] lowmem_reserve[]: 0 0 0 0
> [56317.522626] Node 0 DMA: 56*4kB (UE) 20*8kB (UE) 14*16kB (UE) 21*32kB (UE) 
> 22*64kB (UE) 9*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 
> 7936kB
> [56317.522644] Node 0 DMA32: 254*4kB (UEM) 460*8kB (UEM) 15*16kB (UMR) 0*32kB 
> 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5576kB
> [56317.522660] 9223 total pagecache pages
> [56317.522663] 0 pages in swap cache
> [56317.522668] Swap cache stats: add 0, delete 0, find 0/0
> [56317.522671] Free swap  = 0kB
> [56317.522673] Total swap = 0kB
> [56317.529054] 524287 pages RAM
> [56317.529064] 12090 pages reserved
> [56317.529068] 294744 pages shared
> [56317.529071] 431526 pages non-shared
> [56317.529075] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents 
> oom_score_adj name
> [56317.529090] [  330]     0   330     4213      114      12        0         
> -1000 udevd
> [56317.529098] [  815]     0   815    13018      110      20        0         
>     0 rsyslogd
> [56317.529105] [  853]   104   853    12487      102      27        0         
>     0 zabbix_agentd
> [56317.529112] [  909]     0   909     4212      113      12        0         
> -1000 udevd
> [56317.529119] [  910]     0   910     4212      113      12        0         
> -1000 udevd
> [56317.529126] [  911]   104   911    12487      267      27        0         
>     0 zabbix_agentd
> [56317.529133] [  912]   104   912    12487      101      27        0         
>     0 zabbix_agentd
> [56317.529139] [  913]   104   913    12487      101      27        0         
>     0 zabbix_agentd
> [56317.529146] [  914]   104   914    12487      101      27        0         
>     0 zabbix_agentd
> [56317.529152] [  915]   104   915    13005      109      29        0         
>     0 zabbix_agentd
> [56317.529159] [ 1016]     0  1016    86937     2043     155        0         
>     0 apache2
> [56317.529166] [ 1026]     0  1026     4692       40      14        0         
>     0 atd
> [56317.529177] [ 1054]     0  1054     5747       69      16        0         
>     0 cron
> [56317.529184] [ 1076]   103  1076     5830       54      15        0         
>     0 dbus-daemon
> [56317.529190] [ 1104]   106  1104     9630      153      23        0         
>     0 ntpd
> [56317.529197] [ 1117]     8  1117     2947       45      12        0         
>     0 nullmailer-send
> [56317.529203] [ 1130]     0  1130    18989      395      35        0         
>     0 searchd
> [56317.529211] [ 1140]     0  1140    12307      138      28        0         
> -1000 sshd
> [56317.529220] [ 1152]   105  1152    10265     1088      24        0         
>     0 unbound
> [56317.529228] [ 1172]     0  1172     1496       25       9        0         
>     0 getty
> [56317.529237] [ 1173]     0  1173     1496       24       8        0         
>     0 getty
> [56317.529245] [31557]    33 31557    88162     5277     161        0         
>     0 apache2
> [56317.529252] [ 1031]    33  1031    87879     4726     161        0         
>     0 apache2
> [56317.529261] [ 1579]    33  1579    87493     5027     158        0         
>     0 apache2
> [56317.529267] [ 2901]    33  2901    87622     4707     158        0         
>     0 apache2
> [56317.529274] [ 3447]    33  3447    35706     2055      71        0         
>     0 apache2
> [56317.529281] [ 4559]    33  4559    87714     4945     160        0         
>     0 apache2
> [56317.529287] [ 4560]    33  4560    35648     1981      71        0         
>     0 apache2
> [56317.529296] [ 4561]    33  4561    87859     3840     159        0         
>     0 apache2
> [56317.529304] [ 6205]    33  6205    87422     3053     154        0         
>     0 apache2
> [56317.529311] [ 6776]     0  6776    10460       89      25        0         
>     0 cron
> [56317.529317] [ 6780]     0  6780     1002       23       8        0         
>     0 sh
> [56317.529324] [ 6784]     0  6784     7322      109      20        0         
>     0 wget
> [56317.529331] [ 6847]     0  6847    10460       89      25        0         
>     0 cron
> [56317.529338] [ 6856]     0  6856     1002       23       8        0         
>     0 sh
> [56317.529345] [ 6862]     0  6862     7322      108      20        0         
>     0 wget
> [56317.529351] [ 6924]     0  6924    10460       89      25        0         
>     0 cron
> [56317.529361] [ 6928]     0  6928     1002       24       8        0         
>     0 sh
> [56317.529367] [ 6932]     0  6932     7322      107      20        0         
>     0 wget
> [56317.529374] [ 6948]    33  6948    35279     1532      70        0         
>     0 apache2
> [56317.529381] [ 6991]    33  6991    86937     1966     137        0         
>     0 apache2
> [56317.529388] [ 6992]    33  6992    86937     1966     137        0         
>     0 apache2
> [56317.529394] [ 6993]     0  6993    10460       88      25        0         
>     0 cron
> [56317.529403] [ 6994]     0  6994    10460       88      25        0         
>     0 cron
> [56317.529410] [ 6995]     0  6995    10460       88      25        0         
>     0 cron
> [56317.529416] [ 6999]  1337  6999     1002       23       8        0         
>     0 sh
> [56317.529423] [ 7000]     0  7000     1002       24       8        0         
>     0 sh
> [56317.529429] [ 7001]     0  7001     1002       23       8        0         
>     0 sh
> [56317.529436] [ 7002]  1337  7002     1481       21       9        0         
>     0 sleep
> [56317.529445] [ 7003]     0  7003     7322      110      20        0         
>     0 wget
> [56317.529452] [ 7004]     0  7004     7322      110      20        0         
>     0 wget
> [56317.529458] Out of memory: Kill process 31557 (apache2) score 10 or 
> sacrifice child
> [56317.529464] Killed process 31557 (apache2) total-vm:352648kB, 
> anon-rss:11004kB, file-rss:10104kB
> [56680.628586] apache2 invoked oom-killer: gfp_mask=0x201da, order=0, 
> oom_score_adj=0
> [56680.628601] CPU: 0 PID: 1579 Comm: apache2 Not tainted 2.6.50-dae-xen #2
> [56680.628607]  0000000000000000 ffffffff8134ab12 0000000000000080 
> ffff88006e3ffbb0
> [56680.628636]  0000000000000000 ffffffff81089ad6 ffffffff8134f09a 
> ffff880001a8c8e0
> [56680.628645]  ffffffff814d3d48 0000000000000000 00000000000201da 
> 000000000000000b
> [56680.628654] Call Trace:
> [56680.628666]  [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> [56680.628675]  [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> [56680.628682]  [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> [56680.628692]  [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> [56680.628699]  [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> [56680.628706]  [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> [56680.628713]  [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
> [56680.628720]  [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> [56680.628727]  [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> [56680.628735]  [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> [56680.628741]  [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> [56680.628748]  [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> [56680.628756]  [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> [56680.628763]  [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> [56680.628768] Mem-Info:
> [56680.628771] Node 0 DMA per-cpu:
> [56680.628776] CPU    0: hi:    0, btch:   1 usd:   0
> [56680.628782] Node 0 DMA32 per-cpu:
> [56680.628788] CPU    0: hi:  186, btch:  31 usd: 115
> [56680.628796] active_anon:19702 inactive_anon:1033 isolated_anon:0
> [56680.628796]  active_file:156 inactive_file:1004 isolated_file:0
> [56680.628796]  unevictable:0 dirty:0 writeback:0 unstable:0
> [56680.628796]  free:3382 slab_reclaimable:103079 slab_unreclaimable:233653
> [56680.628796]  mapped:3205 shmem:8327 pagetables:2067 bounce:0
> [56680.628796]  free_cma:0
> [56680.628814] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB 
> active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB 
> managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB 
> slab_reclaimable:1632kB slab_unreclaimable:4156kB kernel_stack:2056kB 
> pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB 
> pages_scanned:0 all_unreclaimable? yes
> [56680.628848] lowmem_reserve[]: 0 1973 1973 1973
> [56680.628856] Node 0 DMA32 free:5600kB min:5660kB low:7072kB high:8488kB 
> active_anon:78780kB inactive_anon:4132kB active_file:624kB 
> inactive_file:4016kB unevictable:0kB isolated(anon):0kB isolated(file):0kB 
> present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB 
> mapped:12820kB shmem:33308kB slab_reclaimable:410684kB 
> slab_unreclaimable:930456kB kernel_stack:576528kB pagetables:8264kB 
> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2646 
> all_unreclaimable? yes
> [56680.628883] lowmem_reserve[]: 0 0 0 0
> [56680.628890] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 10*32kB 
> (EM) 24*64kB (EM) 9*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) 
> = 7928kB
> [56680.628921] Node 0 DMA32: 0*4kB 564*8kB (UEM) 26*16kB (MR) 1*32kB (R) 
> 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5600kB
> [56680.628945] 9494 total pagecache pages
> [56680.628949] 0 pages in swap cache
> [56680.628957] Swap cache stats: add 0, delete 0, find 0/0
> [56680.628962] Free swap  = 0kB
> [56680.628965] Total swap = 0kB
> [56680.634908] 524287 pages RAM
> [56680.634916] 12090 pages reserved
> [56680.634919] 291027 pages shared
> [56680.634921] 431125 pages non-shared
> [56680.634924] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents 
> oom_score_adj name
> [56680.634937] [  330]     0   330     4213      114      12        0         
> -1000 udevd
> [56680.634943] [  815]     0   815    13018      138      20        0         
>     0 rsyslogd
> [56680.634948] [  853]   104   853    12487      102      27        0         
>     0 zabbix_agentd
> [56680.634953] [  909]     0   909     4212      113      12        0         
> -1000 udevd
> [56680.634957] [  910]     0   910     4212      113      12        0         
> -1000 udevd
> [56680.634961] [  911]   104   911    12487      267      27        0         
>     0 zabbix_agentd
> [56680.634966] [  912]   104   912    12487      101      27        0         
>     0 zabbix_agentd
> [56680.634971] [  913]   104   913    12487      101      27        0         
>     0 zabbix_agentd
> [56680.634975] [  914]   104   914    12487      101      27        0         
>     0 zabbix_agentd
> [56680.634980] [  915]   104   915    13005      109      29        0         
>     0 zabbix_agentd
> [56680.634985] [ 1016]     0  1016    86937     2014     155        0         
>     0 apache2
> [56680.634990] [ 1026]     0  1026     4692       40      14        0         
>     0 atd
> [56680.634995] [ 1054]     0  1054     5747       69      16        0         
>     0 cron
> [56680.634999] [ 1076]   103  1076     5830       54      15        0         
>     0 dbus-daemon
> [56680.635004] [ 1104]   106  1104     9630      147      23        0         
>     0 ntpd
> [56680.635008] [ 1117]     8  1117     2947       45      12        0         
>     0 nullmailer-send
> [56680.635013] [ 1130]     0  1130    18989      400      35        0         
>     0 searchd
> [56680.635018] [ 1140]     0  1140    12307      138      28        0         
> -1000 sshd
> [56680.635022] [ 1152]   105  1152    10265     1099      24        0         
>     0 unbound
> [56680.635027] [ 1172]     0  1172     1496       25       9        0         
>     0 getty
> [56680.635144] [ 1173]     0  1173     1496       24       8        0         
>     0 getty
> [56680.635150] [ 1031]    33  1031    87879     4814     161        0         
>     0 apache2
> [56680.635155] [ 1579]    33  1579    87941     5548     159        0         
>     0 apache2
> [56680.635160] [ 2901]    33  2901    87622     4692     158        0         
>     0 apache2
> [56680.635164] [ 4559]    33  4559    87714     4947     160        0         
>     0 apache2
> [56680.635169] [ 4561]    33  4561    87859     3852     159        0         
>     0 apache2
> [56680.635174] [ 6205]    33  6205    87294     3684     155        0         
>     0 apache2
> [56680.635179] [ 6991]    33  6991    87434     3395     152        0         
>     0 apache2
> [56680.635183] [ 6992]    33  6992    87593     3185     148        0         
>     0 apache2
> [56680.635188] [ 7289]     0  7289    10460       89      25        0         
>     0 cron
> [56680.635192] [ 7297]     0  7297     1002       23       7        0         
>     0 sh
> [56680.635197] [ 7303]     0  7303     7322      110      20        0         
>     0 wget
> [56680.635202] [ 7363]     0  7363    10460       89      25        0         
>     0 cron
> [56680.635206] [ 7374]     0  7374     1002       25       7        0         
>     0 sh
> [56680.635213] [ 7378]     0  7378     7322      108      19        0         
>     0 wget
> [56680.635218] [ 7437]     0  7437    10460       89      25        0         
>     0 cron
> [56680.635222] [ 7438]     0  7438    10460       89      25        0         
>     0 cron
> [56680.635227] [ 7439]     0  7439    10460       89      25        0         
>     0 cron
> [56680.635231] [ 7443]     0  7443     1002       23       8        0         
>     0 sh
> [56680.635235] [ 7444]     0  7444     1002       23       8        0         
>     0 sh
> [56680.635239] [ 7445]  1337  7445     1002       24       7        0         
>     0 sh
> [56680.635244] [ 7446]  1337  7446     1481       22       8        0         
>     0 sleep
> [56680.635248] [ 7447]     0  7447     7322      109      19        0         
>     0 wget
> [56680.635252] [ 7448]     0  7448     7322      107      19        0         
>     0 wget
> [56680.635257] Out of memory: Kill process 1579 (apache2) score 11 or 
> sacrifice child
> [56680.635262] Killed process 1579 (apache2) total-vm:351764kB, 
> anon-rss:12092kB, file-rss:10100kB
> [56734.778939] wget invoked oom-killer: gfp_mask=0x201da, order=0, 
> oom_score_adj=0
> [56734.778957] CPU: 0 PID: 7524 Comm: wget Not tainted 2.6.50-dae-xen #2
> [56734.778963]  0000000000000000 ffffffff8134ab12 0000000000000080 
> ffff88006d133bb0
> [56734.778973]  0000000000000000 ffffffff81089ad6 ffffffff8134f09a 
> ffff8800634686a0
> [56734.778982]  ffffffff814d3d48 0000000000000000 00000000000201da 
> 0000000000000009
> [56734.779011] Call Trace:
> [56734.779020]  [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> [56734.779026]  [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> [56734.779031]  [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> [56734.779037]  [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> [56734.779042]  [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> [56734.779046]  [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> [56734.779051]  [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
> [56734.779055]  [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> [56734.779060]  [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> [56734.779065]  [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> [56734.779069]  [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> [56734.779073]  [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> [56734.779079]  [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> [56734.779085]  [<ffffffff81113891>] ? fsnotify+0x261/0x291
> [56734.779089]  [<ffffffff810ed49a>] ? vfs_fstatat+0x3e/0x8d
> [56734.779093]  [<ffffffff810ed50b>] ? SYSC_newstat+0x10/0x27
> [56734.779097]  [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> [56734.779100] Mem-Info:
> [56734.779102] Node 0 DMA per-cpu:
> [56734.779106] CPU    0: hi:    0, btch:   1 usd:   0
> [56734.779108] Node 0 DMA32 per-cpu:
> [56734.779112] CPU    0: hi:  186, btch:  31 usd:  62
> [56734.779117] active_anon:18885 inactive_anon:1033 isolated_anon:0
> [56734.779117]  active_file:190 inactive_file:1109 isolated_file:0
> [56734.779117]  unevictable:0 dirty:2 writeback:0 unstable:0
> [56734.779117]  free:3395 slab_reclaimable:103429 slab_unreclaimable:233857
> [56734.779117]  mapped:3186 shmem:8327 pagetables:2038 bounce:0
> [56734.779117]  free_cma:0
> [56734.779234] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB 
> active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:12kB 
> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB 
> managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB 
> slab_reclaimable:1664kB slab_unreclaimable:4156kB kernel_stack:2056kB 
> pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB 
> pages_scanned:0 all_unreclaimable? yes
> [56734.779256] lowmem_reserve[]: 0 1973 1973 1973
> [56734.779267] Node 0 DMA32 free:5652kB min:5660kB low:7072kB high:8488kB 
> active_anon:75512kB inactive_anon:4132kB active_file:760kB 
> inactive_file:4424kB unevictable:0kB isolated(anon):0kB isolated(file):0kB 
> present:2080768kB managed:2020992kB mlocked:0kB dirty:8kB writeback:0kB 
> mapped:12744kB shmem:33308kB slab_reclaimable:412052kB 
> slab_unreclaimable:931272kB kernel_stack:577184kB pagetables:8148kB 
> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3285 
> all_unreclaimable? yes
> [56734.779293] lowmem_reserve[]: 0 0 0 0
> [56734.779300] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 8*32kB (M) 
> 23*64kB (EM) 8*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 
> 7928kB
> [56734.779327] Node 0 DMA32: 99*4kB (E) 521*8kB (EM) 16*16kB (ER) 6*32kB 
> (UMR) 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 
> 5652kB
> [56734.779351] 9645 total pagecache pages
> [56734.779359] 0 pages in swap cache
> [56734.779363] Swap cache stats: add 0, delete 0, find 0/0
> [56734.779367] Free swap  = 0kB
> [56734.779370] Total swap = 0kB
> [56734.785510] 524287 pages RAM
> [56734.785519] 12090 pages reserved
> [56734.785523] 286620 pages shared
> [56734.785526] 431108 pages non-shared
> [56734.785529] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents 
> oom_score_adj name
> [56734.785544] [  330]     0   330     4213      114      12        0         
> -1000 udevd
> [56734.785553] [  815]     0   815    13018      138      20        0         
>     0 rsyslogd
> [56734.785560] [  853]   104   853    12487      102      27        0         
>     0 zabbix_agentd
> [56734.785567] [  909]     0   909     4212      113      12        0         
> -1000 udevd
> [56734.785573] [  910]     0   910     4212      113      12        0         
> -1000 udevd
> [56734.785579] [  911]   104   911    12487      208      27        0         
>     0 zabbix_agentd
> [56734.785586] [  912]   104   912    12487      101      27        0         
>     0 zabbix_agentd
> [56734.785593] [  913]   104   913    12487      101      27        0         
>     0 zabbix_agentd
> [56734.785600] [  914]   104   914    12487      101      27        0         
>     0 zabbix_agentd
> [56734.785607] [  915]   104   915    13005      109      29        0         
>     0 zabbix_agentd
> [56734.785614] [ 1016]     0  1016    86937     2024     155        0         
>     0 apache2
> [56734.785620] [ 1026]     0  1026     4692       40      14        0         
>     0 atd
> [56734.785627] [ 1054]     0  1054     5747       70      16        0         
>     0 cron
> [56734.785634] [ 1076]   103  1076     5830       54      15        0         
>     0 dbus-daemon
> [56734.785641] [ 1104]   106  1104     9630      147      23        0         
>     0 ntpd
> [56734.785648] [ 1117]     8  1117     2947       45      12        0         
>     0 nullmailer-send
> [56734.785654] [ 1130]     0  1130    18989      395      35        0         
>     0 searchd
> [56734.785661] [ 1140]     0  1140    12307      138      28        0         
> -1000 sshd
> [56734.785665] [ 1152]   105  1152    10265     1099      24        0         
>     0 unbound
> [56734.785670] [ 1172]     0  1172     1496       25       9        0         
>     0 getty
> [56734.785712] [ 1173]     0  1173     1496       24       8        0         
>     0 getty
> [56734.785718] [ 1031]    33  1031    87879     4814     161        0         
>     0 apache2
> [56734.785722] [ 2901]    33  2901    87622     4692     158        0         
>     0 apache2
> [56734.785727] [ 4559]    33  4559    87714     4954     160        0         
>     0 apache2
> [56734.785731] [ 4561]    33  4561    87859     3815     159        0         
>     0 apache2
> [56734.785737] [ 6205]    33  6205    87294     3674     155        0         
>     0 apache2
> [56734.785741] [ 6991]    33  6991    87434     3398     152        0         
>     0 apache2
> [56734.785745] [ 6992]    33  6992    87593     3228     150        0         
>     0 apache2
> [56734.785750] [ 7363]     0  7363    10460       89      25        0         
>     0 cron
> [56734.785754] [ 7374]     0  7374     1002       25       7        0         
>     0 sh
> [56734.785760] [ 7378]     0  7378     7322      108      19        0         
>     0 wget
> [56734.785765] [ 7505]     0  7505    10460       89      25        0         
>     0 cron
> [56734.785769] [ 7506]     0  7506    10460       89      25        0         
>     0 cron
> [56734.785773] [ 7507]     0  7507    10460       89      25        0         
>     0 cron
> [56734.785777] [ 7508]     0  7508    10460       89      25        0         
>     0 cron
> [56734.785782] [ 7509]     0  7509    10460       89      25        0         
>     0 cron
> [56734.785786] [ 7510]     0  7510    10460       89      25        0         
>     0 cron
> [56734.785790] [ 7517]     0  7517     1002       24       8        0         
>     0 sh
> [56734.785794] [ 7518]     0  7518     1002       24       8        0         
>     0 sh
> [56734.785800] [ 7519]     0  7519     1002       24       8        0         
>     0 sh
> [56734.785804] [ 7520]     0  7520     1002       23       8        0         
>     0 sh
> [56734.785808] [ 7521]  1337  7521     1002       24       8        0         
>     0 sh
> [56734.785813] [ 7522]     0  7522     1002       23       7        0         
>     0 sh
> [56734.785817] [ 7523]  1337  7523     1481       21       8        0         
>     0 sleep
> [56734.785822] [ 7524]     0  7524     7322      148      20        0         
>     0 wget
> [56734.785826] [ 7525]     0  7525     7322      111      20        0         
>     0 wget
> [56734.785830] [ 7526]     0  7526     7322      110      19        0         
>     0 wget
> [56734.785834] [ 7527]     0  7527     7322      110      20        0         
>     0 wget
> [56734.785839] [ 7528]     0  7528     1002       24       7        0         
>     0 maxlifetime
> [56734.785846] [ 7530]     0  7530    16356      291      37        0         
>     0 php5
> [56734.785850] Out of memory: Kill process 4559 (apache2) score 9 or 
> sacrifice child
> [56734.785854] Killed process 4559 (apache2) total-vm:350856kB, 
> anon-rss:11596kB, file-rss:8220kB
> [56793.525886] wget invoked oom-killer: gfp_mask=0x201da, order=0, 
> oom_score_adj=0
> [56793.525921] CPU: 0 PID: 7614 Comm: wget Not tainted 2.6.50-dae-xen #2
> [56793.525928]  0000000000000000 ffffffff8134ab12 ffffffff8134f478 
> 00000000000007c5
> [56793.525938]  0000000000000010 ffffffff81089ad6 ffffffff8134f09a 
> ffff880001a127c0
> [56793.525946]  ffffffff814d3d48 0000000000000000 00000000000201da 
> 0000000000000009
> [56793.525976] Call Trace:
> [56793.525986]  [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> [56793.526012]  [<ffffffff8134f478>] ? retint_restore_args+0x5/0x6
> [56793.526042]  [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> [56793.526048]  [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> [56793.526057]  [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> [56793.526064]  [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> [56793.526070]  [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> [56793.526082]  [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
> [56793.526090]  [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> [56793.526097]  [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> [56793.526104]  [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> [56793.526111]  [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> [56793.526117]  [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> [56793.526125]  [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> [56793.526133]  [<ffffffff81113891>] ? fsnotify+0x261/0x291
> [56793.526142]  [<ffffffff810ed49a>] ? vfs_fstatat+0x3e/0x8d
> [56793.526149]  [<ffffffff810ed50b>] ? SYSC_newstat+0x10/0x27
> [56793.526155]  [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> [56793.526160] Mem-Info:
> [56793.526163] Node 0 DMA per-cpu:
> [56793.526171] CPU    0: hi:    0, btch:   1 usd:   0
> [56793.526175] Node 0 DMA32 per-cpu:
> [56793.526180] CPU    0: hi:  186, btch:  31 usd: 150
> [56793.526188] active_anon:17848 inactive_anon:1033 isolated_anon:5
> [56793.526188]  active_file:78 inactive_file:778 isolated_file:0
> [56793.526188]  unevictable:0 dirty:0 writeback:0 unstable:0
> [56793.526188]  free:3395 slab_reclaimable:104385 slab_unreclaimable:234096
> [56793.526188]  mapped:2974 shmem:8327 pagetables:2047 bounce:0
> [56793.526188]  free_cma:0
> [56793.526213] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB 
> active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:12kB 
> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB 
> managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB 
> slab_reclaimable:1664kB slab_unreclaimable:4156kB kernel_stack:2056kB 
> pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB 
> pages_scanned:3 all_unreclaimable? yes
> [56793.526238] lowmem_reserve[]: 0 1973 1973 1973
> [56793.526247] Node 0 DMA32 free:5652kB min:5660kB low:7072kB high:8488kB 
> active_anon:71364kB inactive_anon:4132kB active_file:312kB 
> inactive_file:3100kB unevictable:0kB isolated(anon):20kB isolated(file):0kB 
> present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB 
> mapped:11896kB shmem:33308kB slab_reclaimable:415876kB 
> slab_unreclaimable:932228kB kernel_stack:577872kB pagetables:8184kB 
> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1295 
> all_unreclaimable? yes
> [56793.526277] lowmem_reserve[]: 0 0 0 0
> [56793.526284] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 8*32kB (M) 
> 23*64kB (EM) 8*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 
> 7928kB
> [56793.526316] Node 0 DMA32: 3*4kB (UEM) 621*8kB (UEM) 2*16kB (UR) 0*32kB 
> 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5652kB
> [56793.526343] 9197 total pagecache pages
> [56793.526347] 0 pages in swap cache
> [56793.526352] Swap cache stats: add 0, delete 0, find 0/0
> [56793.526356] Free swap  = 0kB
> [56793.526359] Total swap = 0kB
> [56793.532348] 524287 pages RAM
> [56793.532355] 12090 pages reserved
> [56793.532358] 284892 pages shared
> [56793.532360] 430988 pages non-shared
> [56793.532363] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents 
> oom_score_adj name
> [56793.532376] [  330]     0   330     4213      114      12        0         
> -1000 udevd
> [56793.532381] [  815]     0   815    13018      138      20        0         
>     0 rsyslogd
> [56793.532386] [  853]   104   853    12487      102      27        0         
>     0 zabbix_agentd
> [56793.532391] [  909]     0   909     4212      113      12        0         
> -1000 udevd
> [56793.532396] [  910]     0   910     4212      113      12        0         
> -1000 udevd
> [56793.532400] [  911]   104   911    12487      207      27        0         
>     0 zabbix_agentd
> [56793.532405] [  912]   104   912    12487      101      27        0         
>     0 zabbix_agentd
> [56793.532409] [  913]   104   913    12487      101      27        0         
>     0 zabbix_agentd
> [56793.532414] [  914]   104   914    12487      101      27        0         
>     0 zabbix_agentd
> [56793.532418] [  915]   104   915    13005      109      29        0         
>     0 zabbix_agentd
> [56793.532424] [ 1016]     0  1016    86937     2028     155        0         
>     0 apache2
> [56793.532428] [ 1026]     0  1026     4692       40      14        0         
>     0 atd
> [56793.532433] [ 1054]     0  1054     5747       70      16        0         
>     0 cron
> [56793.532437] [ 1076]   103  1076     5830       54      15        0         
>     0 dbus-daemon
> [56793.532442] [ 1104]   106  1104     9630      147      23        0         
>     0 ntpd
> [56793.532447] [ 1117]     8  1117     2947       45      12        0         
>     0 nullmailer-send
> [56793.532451] [ 1130]     0  1130    18989      374      35        0         
>     0 searchd
> [56793.532456] [ 1140]     0  1140    12307      138      28        0         
> -1000 sshd
> [56793.532461] [ 1152]   105  1152    10265     1099      24        0         
>     0 unbound
> [56793.532467] [ 1172]     0  1172     1496       25       9        0         
>     0 getty
> [56793.532484] [ 1173]     0  1173     1496       24       8        0         
>     0 getty
> [56793.532491] [ 1031]    33  1031    87879     4791     161        0         
>     0 apache2
> [56793.532497] [ 2901]    33  2901    87622     4723     158        0         
>     0 apache2
> [56793.532504] [ 4561]    33  4561    87859     3866     159        0         
>     0 apache2
> [56793.532510] [ 6205]    33  6205    87294     3663     155        0         
>     0 apache2
> [56793.532515] [ 6991]    33  6991    87434     3398     152        0         
>     0 apache2
> [56793.532522] [ 6992]    33  6992    87593     3228     150        0         
>     0 apache2
> [56793.532526] [ 7363]     0  7363    10460       89      25        0         
>     0 cron
> [56793.532531] [ 7374]     0  7374     1002       25       7        0         
>     0 sh
> [56793.532537] [ 7378]     0  7378     7322      108      19        0         
>     0 wget
> [56793.532547] [ 7555]    33  7555    86937     1967     137        0         
>     0 apache2
> [56793.532553] [ 7593]     0  7593    10460       88      25        0         
>     0 cron
> [56793.532559] [ 7594]     0  7594    10460       88      25        0         
>     0 cron
> [56793.532564] [ 7595]     0  7595    10460       88      25        0         
>     0 cron
> [56793.532568] [ 7596]     0  7596    10460       88      25        0         
>     0 cron
> [56793.532573] [ 7597]     0  7597    10460       88      25        0         
>     0 cron
> [56793.532580] [ 7598]     0  7598    10460       88      25        0         
>     0 cron
> [56793.532584] [ 7599]     0  7599    10460       88      25        0         
>     0 cron
> [56793.532588] [ 7607]     0  7607     1002       25       8        0         
>     0 sh
> [56793.532593] [ 7608]  1337  7608     1002       24       8        0         
>     0 sh
> [56793.532597] [ 7609]     0  7609     1002       26       7        0         
>     0 sh
> [56793.532601] [ 7610]     0  7610     1002       24       7        0         
>     0 sh
> [56793.532606] [ 7611]     0  7611     1002       25       7        0         
>     0 sh
> [56793.532611] [ 7612]     0  7612     1002       23       8        0         
>     0 sh
> [56793.532615] [ 7613]     0  7613     1002       25       7        0         
>     0 sh
> [56793.532620] [ 7614]     0  7614     7322      156      20        0         
>     0 wget
> [56793.532624] [ 7615]  1337  7615     1481       19       8        0         
>     0 sleep
> [56793.532628] [ 7616]     0  7616     7322      110      19        0         
>     0 wget
> [56793.532632] [ 7617]     0  7617     7322      110      20        0         
>     0 wget
> [56793.532637] [ 7618]     0  7618     7322      109      19        0         
>     0 wget
> [56793.532641] [ 7619]     0  7619     7322      109      20        0         
>     0 wget
> [56793.532646] [ 7620]     0  7620     7322      109      20        0         
>     0 wget
> [56793.532652] Out of memory: Kill process 1031 (apache2) score 9 or 
> sacrifice child
> [56793.532656] Killed process 1031 (apache2) total-vm:351516kB, 
> anon-rss:9624kB, file-rss:9540kB
> 
> ...
> 
> [57131.763804] zabbix_sender invoked oom-killer: gfp_mask=0x201da, order=0, 
> oom_score_adj=0
> [57131.763840] CPU: 0 PID: 8158 Comm: zabbix_sender Not tainted 
> 2.6.50-dae-xen #2
> [57131.763847]  0000000000000000 ffffffff8134ab12 0000000000000080 
> ffff880073e21bb0
> [57131.763858]  0000000000000000 ffffffff81089ad6 ffffffff8134f09a 
> ffff8800788aea00
> [57131.763867]  ffffffff814d3d48 0000000000000000 00000000000201da 
> 0000000000000004
> [57131.763885] Call Trace:
> [57131.763896]  [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193
> [57131.763906]  [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8
> [57131.763913]  [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13
> [57131.763921]  [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca
> [57131.763947]  [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d
> [57131.763955]  [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666
> [57131.763962]  [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f
> [57131.763969]  [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d
> [57131.763977]  [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e
> [57131.763985]  [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73
> [57131.763991]  [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed
> [57131.763997]  [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2
> [57131.764006]  [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324
> [57131.764017]  [<ffffffff810cbcbc>] ? __split_vma.isra.25+0x11a/0x182
> [57131.764023]  [<ffffffff810ce298>] ? mprotect_fixup+0x1bd/0x1f7
> [57131.764030]  [<ffffffff8134f6e5>] ? page_fault+0x25/0x30
> [57131.764034] Mem-Info:
> [57131.764037] Node 0 DMA per-cpu:
> [57131.764041] CPU    0: hi:    0, btch:   1 usd:   0
> [57131.764045] Node 0 DMA32 per-cpu:
> [57131.764049] CPU    0: hi:  186, btch:  31 usd:  62
> [57131.764076] active_anon:11743 inactive_anon:1026 isolated_anon:0
> [57131.764076]  active_file:288 inactive_file:1121 isolated_file:0
> [57131.764076]  unevictable:0 dirty:0 writeback:0 unstable:0
> [57131.764076]  free:3369 slab_reclaimable:109088 slab_unreclaimable:235420
> [57131.764076]  mapped:358 shmem:8328 pagetables:821 bounce:0
> [57131.764076]  free_cma:0
> [57131.764092] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB 
> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:12kB 
> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB 
> managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB 
> slab_reclaimable:1760kB slab_unreclaimable:4152kB kernel_stack:2056kB 
> pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB 
> pages_scanned:0 all_unreclaimable? yes
> [57131.764115] lowmem_reserve[]: 0 1973 1973 1973
> [57131.764122] Node 0 DMA32 free:5548kB min:5660kB low:7072kB high:8488kB 
> active_anon:46972kB inactive_anon:4104kB active_file:1152kB 
> inactive_file:4472kB unevictable:0kB isolated(anon):0kB isolated(file):0kB 
> present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB 
> mapped:1432kB shmem:33312kB slab_reclaimable:434592kB 
> slab_unreclaimable:937528kB kernel_stack:582224kB pagetables:3284kB 
> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4240 
> all_unreclaimable? yes
> [57131.764151] lowmem_reserve[]: 0 0 0 0
> [57131.764158] Node 0 DMA: 64*4kB (UE) 33*8kB (UEM) 13*16kB (UEM) 11*32kB 
> (UEM) 19*64kB (EM) 8*128kB (M) 0*256kB 1*512kB (M) 0*1024kB 0*2048kB 1*4096kB 
> (R) = 7928kB
> [57131.764189] Node 0 DMA32: 383*4kB (UEM) 164*8kB (UEM) 79*16kB (UMR) 
> 17*32kB (UEMR) 4*64kB (EM) 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 
> 0*4096kB = 5548kB
> [57131.764242] 9744 total pagecache pages
> [57131.764253] 0 pages in swap cache
> [57131.764256] Swap cache stats: add 0, delete 0, find 0/0
> [57131.764263] Free swap  = 0kB
> [57131.764267] Total swap = 0kB
> [57131.770252] 524287 pages RAM
> [57131.770260] 12090 pages reserved
> [57131.770262] 265440 pages shared
> [57131.770265] 433233 pages non-shared
> [57131.770267] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents 
> oom_score_adj name
> [57131.770280] [  330]     0   330     4213      114      12        0         
> -1000 udevd
> [57131.770285] [  815]     0   815    13048      150      20        0         
>     0 rsyslogd
> [57131.770290] [  853]   104   853    12486       99      27        0         
>     0 zabbix_agentd
> [57131.770295] [  909]     0   909     4212      113      12        0         
> -1000 udevd
> [57131.770299] [  910]     0   910     4212      113      12        0         
> -1000 udevd
> [57131.770305] [ 1016]     0  1016    86937     2031     155        0         
>     0 apache2
> [57131.770310] [ 1054]     0  1054     5747       69      16        0         
>     0 cron
> [57131.770314] [ 1076]   103  1076     5830       54      15        0         
>     0 dbus-daemon
> [57131.770319] [ 1104]   106  1104     9630      147      23        0         
>     0 ntpd
> [57131.770323] [ 1130]     0  1130    18989      374      35        0         
>     0 searchd
> [57131.770328] [ 1140]     0  1140    12307      138      28        0         
> -1000 sshd
> [57131.770334] [ 7363]     0  7363    10460       89      25        0         
>     0 cron
> [57131.770339] [ 7374]     0  7374     1002       25       7        0         
>     0 sh
> [57131.770343] [ 7378]     0  7378     7322      108      19        0         
>     0 wget
> [57131.770348] [ 8003]     0  8003    10460       88      25        0         
>     0 cron
> [57131.770352] [ 8006]     0  8006    10460       88      25        0         
>     0 cron
> [57131.770357] [ 8009]     0  8009    10460       88      25        0         
>     0 cron
> [57131.770361] [ 8020]  1337  8020     1002       23       7        0         
>     0 sh
> [57131.770366] [ 8023]     0  8023     1002       23       6        0         
>     0 sh
> [57131.770370] [ 8024]     0  8024     1002       24       7        0         
>     0 sh
> [57131.770399] [ 8030]     0  8030     7322      108      19        0         
>     0 wget
> [57131.770406] [ 8035]     0  8035     7322      109      20        0         
>     0 wget
> [57131.770410] [ 8051]     0  8051     1002       41       8        0         
>     0 rc
> [57131.770414] [ 8054]     0  8054     3085       60      11        0         
>     0 startpar
> [57131.770419] [ 8055]  1337  8055     1002       23       8        0         
>     0 zabbix-launch-t
> [57131.770424] [ 8057]     0  8057     1002       40       7        0         
>     0 apache2
> [57131.770430] [ 8067]  1337  8067     2841       76       9        0         
>     0 apache
> [57131.770436] [ 8084]     0  8084     1002       24       8        0         
>     0 apache2ctl
> [57131.770443] [ 8092]     0  8092    26561      288      48        0         
>     0 apache2
> [57131.770450] [ 8115]     0  8115     1002       22       8        0         
>     0 zabbix-agent
> [57131.770457] [ 8116]     0  8116      980       32       8        0         
>     0 start-stop-daem
> [57131.770461] [ 8158]  1337  8158    10130      168      22        0         
>     0 zabbix_sender
> [57131.770466] [ 8161]    33  8161    86937     1967     137        0         
>     0 apache2
> [57131.770471] Out of memory: Kill process 8161 (apache2) score 4 or 
> sacrifice child
> [57131.770475] Killed process 8161 (apache2) total-vm:347748kB, 
> anon-rss:7868kB, file-rss:0kB
> 
> 
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxx
> http://lists.xen.org/xen-users



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.