|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-devel] unexpected Out Of Memory (OOM)
Hi, I have a memory problem on several PV domU, I *suppose* since the upgrade to 3.10.4 kernel (and still occur with 3.10.5). I use Xen 4.2.2 (from Debian sid), and the dom0 run under Debian Wheezy with a vanilla 3.9.11 kernel. The problem : once a domU throw oom-killer, then it loop until all processes will be killed, then the kernel abort and shutdown the VM. I'm not sure how to read the OOM-killer messages about memory state, so I copy/paste a list of them here, about a 2GB RAM domU : [56313.921381] cron invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 [56313.921407] CPU: 0 PID: 7001 Comm: cron Not tainted 2.6.50-dae-xen #2 [56313.921413] 0000000000000000 ffffffff8134ab12 ffff880079cd2120 0000000000000000 [56313.921424] 00000000000280da ffffffff81089ad6 ffffffff8134f09a ffff88007773bba0 [56313.921433] ffffffff814d3d48 0000000000000000 00000000000280da 000000000000000b [56313.921443] Call Trace: [56313.921454] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193 [56313.921462] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8 [56313.921470] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13 [56313.921480] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca [56313.921486] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d [56313.921493] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666 [56313.921502] [<ffffffff810da1db>] ? alloc_pages_vma+0xd4/0xfc [56313.921510] [<ffffffff810cf31f>] ? anon_vma_alloc+0x11/0x21 [56313.921516] [<ffffffff810c7ff1>] ? handle_pte_fault+0x60f/0x6ed [56313.921538] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2 [56313.921547] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324 [56313.921560] [<ffffffff81072f4b>] ? lg_local_lock+0x11/0x14 [56313.921566] [<ffffffff810fee0e>] ? mntput_no_expire+0x2d/0x133 [56313.921572] [<ffffffff81072f5f>] ? lg_local_unlock+0x11/0x14 [56313.921579] [<ffffffff810f3dca>] ? do_last+0x8eb/0x9c3 [56313.921587] [<ffffffff811b0401>] ? string.isra.5+0x37/0x9b [56313.921593] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30 [56313.921600] [<ffffffff81101c75>] ? seq_read+0xb8/0x31a [56313.921606] [<ffffffff811b185d>] ? copy_user_generic_string+0x2d/0x40 [56313.921612] [<ffffffff81101e60>] ? seq_read+0x2a3/0x31a [56313.921632] [<ffffffff81129c95>] ? proc_reg_read+0x54/0x74 [56313.921644] [<ffffffff810e977f>] ? vfs_read+0x8e/0xf0 [56313.921650] [<ffffffff810e9e38>] ? SyS_read+0x41/0x70 [56313.921656] [<ffffffff8134fc69>] ? system_call_fastpath+0x16/0x1b [56313.921662] Mem-Info: [56313.921671] Node 0 DMA per-cpu: [56313.921676] CPU 0: hi: 0, btch: 1 usd: 0 [56313.921680] Node 0 DMA32 per-cpu: [56313.921684] CPU 0: hi: 186, btch: 31 usd: 83 [56313.921693] active_anon:22546 inactive_anon:1033 isolated_anon:1 [56313.921693] active_file:153 inactive_file:849 isolated_file:0 [56313.921693] unevictable:0 dirty:0 writeback:0 unstable:0 [56313.921693] free:3392 slab_reclaimable:101577 slab_unreclaimable:232545 [56313.921693] mapped:3613 shmem:8327 pagetables:2848 bounce:0 [56313.921693] free_cma:0 [56313.921708] Node 0 DMA free:7936kB min:44kB low:52kB high:64kB active_anon:132kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1568kB slab_unreclaimable:3820kB kernel_stack:2056kB pagetables:360kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [56313.921726] lowmem_reserve[]: 0 1973 1973 1973 [56313.921732] Node 0 DMA32 free:5632kB min:5660kB low:7072kB high:8488kB active_anon:90052kB inactive_anon:4132kB active_file:612kB inactive_file:3396kB unevictable:0kB isolated(anon):4kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:14452kB shmem:33308kB slab_reclaimable:404740kB slab_unreclaimable:926360kB kernel_stack:572928kB pagetables:11032kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2541 all_unreclaimable? yes [56313.921749] lowmem_reserve[]: 0 0 0 0 [56313.921757] Node 0 DMA: 56*4kB (UE) 20*8kB (UE) 14*16kB (UE) 21*32kB (UE) 22*64kB (UE) 9*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7936kB [56313.921775] Node 0 DMA32: 204*4kB (EM) 492*8kB (EM) 15*16kB (UMR) 0*32kB 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5632kB [56313.921791] 9341 total pagecache pages [56313.921794] 0 pages in swap cache [56313.921797] Swap cache stats: add 0, delete 0, find 0/0 [56313.921799] Free swap = 0kB [56313.921802] Total swap = 0kB [56313.927968] 524287 pages RAM [56313.927976] 12090 pages reserved [56313.927978] 307132 pages shared [56313.927980] 430905 pages non-shared [56313.927983] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [56313.927996] [ 330] 0 330 4213 114 12 0 -1000 udevd [56313.928001] [ 815] 0 815 13018 114 20 0 0 rsyslogd [56313.928006] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd [56313.928011] [ 909] 0 909 4212 113 12 0 -1000 udevd [56313.928015] [ 910] 0 910 4212 113 12 0 -1000 udevd [56313.928020] [ 911] 104 911 12487 207 27 0 0 zabbix_agentd [56313.928025] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd [56313.928029] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd [56313.928034] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd [56313.928038] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd [56313.928043] [ 1016] 0 1016 86937 2033 155 0 0 apache2 [56313.928048] [ 1026] 0 1026 4692 40 14 0 0 atd [56313.928057] [ 1054] 0 1054 5747 69 16 0 0 cron [56313.928062] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon [56313.928067] [ 1104] 106 1104 9630 147 23 0 0 ntpd [56313.928072] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send [56313.928077] [ 1130] 0 1130 18989 374 35 0 0 searchd [56313.928081] [ 1140] 0 1140 12307 138 28 0 -1000 sshd [56313.928086] [ 1152] 105 1152 10265 1088 24 0 0 unbound [56313.928090] [ 1172] 0 1172 1496 25 9 0 0 getty [56313.928150] [ 1173] 0 1173 1496 24 8 0 0 getty [56313.928157] [31557] 33 31557 88162 5277 161 0 0 apache2 [56313.928163] [31998] 33 31998 88044 5689 162 0 0 apache2 [56313.928168] [ 1031] 33 1031 87879 4726 161 0 0 apache2 [56313.928172] [ 1579] 33 1579 87493 5027 158 0 0 apache2 [56313.928177] [ 2901] 33 2901 87622 4707 158 0 0 apache2 [56313.928182] [ 3447] 33 3447 82447 4550 150 0 0 apache2 [56313.928186] [ 4559] 33 4559 87714 4945 160 0 0 apache2 [56313.928191] [ 4560] 33 4560 82914 4906 150 0 0 apache2 [56313.928195] [ 4561] 33 4561 87859 3840 159 0 0 apache2 [56313.928200] [ 6205] 33 6205 87422 3053 154 0 0 apache2 [56313.928204] [ 6776] 0 6776 10460 89 25 0 0 cron [56313.928229] [ 6780] 0 6780 1002 23 8 0 0 sh [56313.928236] [ 6784] 0 6784 7322 109 20 0 0 wget [56313.928241] [ 6847] 0 6847 10460 89 25 0 0 cron [56313.928246] [ 6856] 0 6856 1002 23 8 0 0 sh [56313.928253] [ 6862] 0 6862 7322 108 20 0 0 wget [56313.928258] [ 6924] 0 6924 10460 89 25 0 0 cron [56313.928263] [ 6928] 0 6928 1002 24 8 0 0 sh [56313.928267] [ 6932] 0 6932 7322 107 20 0 0 wget [56313.928272] [ 6948] 33 6948 82020 2026 135 0 0 apache2 [56313.928277] [ 6991] 33 6991 86937 1966 137 0 0 apache2 [56313.928281] [ 6992] 33 6992 86937 1966 137 0 0 apache2 [56313.928285] [ 6993] 0 6993 10460 106 25 0 0 cron [56313.928290] [ 6994] 0 6994 10460 106 25 0 0 cron [56313.928294] [ 6995] 0 6995 10460 106 25 0 0 cron [56313.928298] [ 6999] 1337 6999 10459 183 23 0 0 cron [56313.928302] [ 7000] 0 7000 10459 182 23 0 0 cron [56313.928306] [ 7001] 0 7001 10460 182 23 0 0 cron [56313.928311] Out of memory: Kill process 31998 (apache2) score 11 or sacrifice child [56313.928315] Killed process 31998 (apache2) total-vm:352176kB, anon-rss:10848kB, file-rss:11908kB [56317.522091] searchd invoked oom-killer: gfp_mask=0x2840d0, order=0, oom_score_adj=0 [56317.522109] CPU: 0 PID: 1130 Comm: searchd Not tainted 2.6.50-dae-xen #2 [56317.522115] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff8800793078e8 [56317.522125] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff880001802120 [56317.522133] ffffffff814d3d48 0000000000000000 00000000002840d0 000000000000000a [56317.522142] Call Trace: [56317.522153] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193 [56317.522161] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8 [56317.522168] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13 [56317.522177] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca [56317.522184] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d [56317.522190] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666 [56317.522197] [<ffffffff810deff1>] ? new_slab+0x6e/0x1d5 [56317.522203] [<ffffffff8134b905>] ? __slab_alloc+0x28f/0x38b [56317.522211] [<ffffffff811ac4bc>] ? radix_tree_preload+0x30/0x77 [56317.522219] [<ffffffff81111258>] ? mpage_alloc+0x25/0x7d [56317.522226] [<ffffffff81140128>] ? _ext4_get_block+0x141/0x141 [56317.522232] [<ffffffff810e00b4>] ? kmem_cache_alloc+0x3d/0x8f [56317.522238] [<ffffffff811ac4bc>] ? radix_tree_preload+0x30/0x77 [56317.522245] [<ffffffff810ae675>] ? add_to_page_cache_locked+0x1b/0x92 [56317.522251] [<ffffffff810ae6fa>] ? add_to_page_cache_lru+0xe/0x31 [56317.522258] [<ffffffff8111185f>] ? mpage_readpages+0x9c/0x105 [56317.522264] [<ffffffff81140128>] ? _ext4_get_block+0x141/0x141 [56317.522271] [<ffffffff810b6aeb>] ? __do_page_cache_readahead+0x112/0x1ab [56317.522277] [<ffffffff810b6ddb>] ? ra_submit+0x18/0x1b [56317.522283] [<ffffffff810b0309>] ? filemap_fault+0x173/0x35f [56317.522289] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d [56317.522296] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e [56317.522304] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73 [56317.522310] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed [56317.522440] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2 [56317.522449] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324 [56317.522457] [<ffffffff81113891>] ? fsnotify+0x261/0x291 [56317.522464] [<ffffffff81350efb>] ? xen_hypervisor_callback+0x1b/0x20 [56317.522471] [<ffffffff81005319>] ? xen_clocksource_read+0x1c/0x1d [56317.522479] [<ffffffff8107f3fc>] ? timekeeping_get_ns.constprop.10+0xd/0x31 [56317.522498] [<ffffffff8107f778>] ? ktime_get_ts+0x35/0x72 [56317.522503] [<ffffffff810f6d92>] ? poll_select_copy_remaining+0xde/0xf3 [56317.522508] [<ffffffff81005319>] ? xen_clocksource_read+0x1c/0x1d [56317.522512] [<ffffffff8107f3fc>] ? timekeeping_get_ns.constprop.10+0xd/0x31 [56317.522517] [<ffffffff8107f8e7>] ? __getnstimeofday+0x1b/0x55 [56317.522522] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30 [56317.522526] Mem-Info: [56317.522529] Node 0 DMA per-cpu: [56317.522540] CPU 0: hi: 0, btch: 1 usd: 0 [56317.522544] Node 0 DMA32 per-cpu: [56317.522549] CPU 0: hi: 186, btch: 31 usd: 90 [56317.522556] active_anon:22815 inactive_anon:1033 isolated_anon:0 [56317.522556] active_file:95 inactive_file:793 isolated_file:0 [56317.522556] unevictable:0 dirty:0 writeback:0 unstable:0 [56317.522556] free:3378 slab_reclaimable:101712 slab_unreclaimable:232523 [56317.522556] mapped:3140 shmem:8327 pagetables:2478 bounce:0 [56317.522556] free_cma:0 [56317.522572] Node 0 DMA free:7936kB min:44kB low:52kB high:64kB active_anon:132kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1568kB slab_unreclaimable:3820kB kernel_stack:2056kB pagetables:360kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [56317.522590] lowmem_reserve[]: 0 1973 1973 1973 [56317.522599] Node 0 DMA32 free:5576kB min:5660kB low:7072kB high:8488kB active_anon:91128kB inactive_anon:4132kB active_file:380kB inactive_file:3172kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:12560kB shmem:33308kB slab_reclaimable:405280kB slab_unreclaimable:926272kB kernel_stack:572976kB pagetables:9552kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3288 all_unreclaimable? yes [56317.522621] lowmem_reserve[]: 0 0 0 0 [56317.522626] Node 0 DMA: 56*4kB (UE) 20*8kB (UE) 14*16kB (UE) 21*32kB (UE) 22*64kB (UE) 9*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7936kB [56317.522644] Node 0 DMA32: 254*4kB (UEM) 460*8kB (UEM) 15*16kB (UMR) 0*32kB 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5576kB [56317.522660] 9223 total pagecache pages [56317.522663] 0 pages in swap cache [56317.522668] Swap cache stats: add 0, delete 0, find 0/0 [56317.522671] Free swap = 0kB [56317.522673] Total swap = 0kB [56317.529054] 524287 pages RAM [56317.529064] 12090 pages reserved [56317.529068] 294744 pages shared [56317.529071] 431526 pages non-shared [56317.529075] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [56317.529090] [ 330] 0 330 4213 114 12 0 -1000 udevd [56317.529098] [ 815] 0 815 13018 110 20 0 0 rsyslogd [56317.529105] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd [56317.529112] [ 909] 0 909 4212 113 12 0 -1000 udevd [56317.529119] [ 910] 0 910 4212 113 12 0 -1000 udevd [56317.529126] [ 911] 104 911 12487 267 27 0 0 zabbix_agentd [56317.529133] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd [56317.529139] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd [56317.529146] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd [56317.529152] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd [56317.529159] [ 1016] 0 1016 86937 2043 155 0 0 apache2 [56317.529166] [ 1026] 0 1026 4692 40 14 0 0 atd [56317.529177] [ 1054] 0 1054 5747 69 16 0 0 cron [56317.529184] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon [56317.529190] [ 1104] 106 1104 9630 153 23 0 0 ntpd [56317.529197] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send [56317.529203] [ 1130] 0 1130 18989 395 35 0 0 searchd [56317.529211] [ 1140] 0 1140 12307 138 28 0 -1000 sshd [56317.529220] [ 1152] 105 1152 10265 1088 24 0 0 unbound [56317.529228] [ 1172] 0 1172 1496 25 9 0 0 getty [56317.529237] [ 1173] 0 1173 1496 24 8 0 0 getty [56317.529245] [31557] 33 31557 88162 5277 161 0 0 apache2 [56317.529252] [ 1031] 33 1031 87879 4726 161 0 0 apache2 [56317.529261] [ 1579] 33 1579 87493 5027 158 0 0 apache2 [56317.529267] [ 2901] 33 2901 87622 4707 158 0 0 apache2 [56317.529274] [ 3447] 33 3447 35706 2055 71 0 0 apache2 [56317.529281] [ 4559] 33 4559 87714 4945 160 0 0 apache2 [56317.529287] [ 4560] 33 4560 35648 1981 71 0 0 apache2 [56317.529296] [ 4561] 33 4561 87859 3840 159 0 0 apache2 [56317.529304] [ 6205] 33 6205 87422 3053 154 0 0 apache2 [56317.529311] [ 6776] 0 6776 10460 89 25 0 0 cron [56317.529317] [ 6780] 0 6780 1002 23 8 0 0 sh [56317.529324] [ 6784] 0 6784 7322 109 20 0 0 wget [56317.529331] [ 6847] 0 6847 10460 89 25 0 0 cron [56317.529338] [ 6856] 0 6856 1002 23 8 0 0 sh [56317.529345] [ 6862] 0 6862 7322 108 20 0 0 wget [56317.529351] [ 6924] 0 6924 10460 89 25 0 0 cron [56317.529361] [ 6928] 0 6928 1002 24 8 0 0 sh [56317.529367] [ 6932] 0 6932 7322 107 20 0 0 wget [56317.529374] [ 6948] 33 6948 35279 1532 70 0 0 apache2 [56317.529381] [ 6991] 33 6991 86937 1966 137 0 0 apache2 [56317.529388] [ 6992] 33 6992 86937 1966 137 0 0 apache2 [56317.529394] [ 6993] 0 6993 10460 88 25 0 0 cron [56317.529403] [ 6994] 0 6994 10460 88 25 0 0 cron [56317.529410] [ 6995] 0 6995 10460 88 25 0 0 cron [56317.529416] [ 6999] 1337 6999 1002 23 8 0 0 sh [56317.529423] [ 7000] 0 7000 1002 24 8 0 0 sh [56317.529429] [ 7001] 0 7001 1002 23 8 0 0 sh [56317.529436] [ 7002] 1337 7002 1481 21 9 0 0 sleep [56317.529445] [ 7003] 0 7003 7322 110 20 0 0 wget [56317.529452] [ 7004] 0 7004 7322 110 20 0 0 wget [56317.529458] Out of memory: Kill process 31557 (apache2) score 10 or sacrifice child [56317.529464] Killed process 31557 (apache2) total-vm:352648kB, anon-rss:11004kB, file-rss:10104kB [56680.628586] apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [56680.628601] CPU: 0 PID: 1579 Comm: apache2 Not tainted 2.6.50-dae-xen #2 [56680.628607] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff88006e3ffbb0 [56680.628636] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff880001a8c8e0 [56680.628645] ffffffff814d3d48 0000000000000000 00000000000201da 000000000000000b [56680.628654] Call Trace: [56680.628666] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193 [56680.628675] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8 [56680.628682] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13 [56680.628692] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca [56680.628699] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d [56680.628706] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666 [56680.628713] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f [56680.628720] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d [56680.628727] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e [56680.628735] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73 [56680.628741] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed [56680.628748] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2 [56680.628756] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324 [56680.628763] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30 [56680.628768] Mem-Info: [56680.628771] Node 0 DMA per-cpu: [56680.628776] CPU 0: hi: 0, btch: 1 usd: 0 [56680.628782] Node 0 DMA32 per-cpu: [56680.628788] CPU 0: hi: 186, btch: 31 usd: 115 [56680.628796] active_anon:19702 inactive_anon:1033 isolated_anon:0 [56680.628796] active_file:156 inactive_file:1004 isolated_file:0 [56680.628796] unevictable:0 dirty:0 writeback:0 unstable:0 [56680.628796] free:3382 slab_reclaimable:103079 slab_unreclaimable:233653 [56680.628796] mapped:3205 shmem:8327 pagetables:2067 bounce:0 [56680.628796] free_cma:0 [56680.628814] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1632kB slab_unreclaimable:4156kB kernel_stack:2056kB pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [56680.628848] lowmem_reserve[]: 0 1973 1973 1973 [56680.628856] Node 0 DMA32 free:5600kB min:5660kB low:7072kB high:8488kB active_anon:78780kB inactive_anon:4132kB active_file:624kB inactive_file:4016kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:12820kB shmem:33308kB slab_reclaimable:410684kB slab_unreclaimable:930456kB kernel_stack:576528kB pagetables:8264kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2646 all_unreclaimable? yes [56680.628883] lowmem_reserve[]: 0 0 0 0 [56680.628890] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 10*32kB (EM) 24*64kB (EM) 9*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB [56680.628921] Node 0 DMA32: 0*4kB 564*8kB (UEM) 26*16kB (MR) 1*32kB (R) 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5600kB [56680.628945] 9494 total pagecache pages [56680.628949] 0 pages in swap cache [56680.628957] Swap cache stats: add 0, delete 0, find 0/0 [56680.628962] Free swap = 0kB [56680.628965] Total swap = 0kB [56680.634908] 524287 pages RAM [56680.634916] 12090 pages reserved [56680.634919] 291027 pages shared [56680.634921] 431125 pages non-shared [56680.634924] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [56680.634937] [ 330] 0 330 4213 114 12 0 -1000 udevd [56680.634943] [ 815] 0 815 13018 138 20 0 0 rsyslogd [56680.634948] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd [56680.634953] [ 909] 0 909 4212 113 12 0 -1000 udevd [56680.634957] [ 910] 0 910 4212 113 12 0 -1000 udevd [56680.634961] [ 911] 104 911 12487 267 27 0 0 zabbix_agentd [56680.634966] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd [56680.634971] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd [56680.634975] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd [56680.634980] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd [56680.634985] [ 1016] 0 1016 86937 2014 155 0 0 apache2 [56680.634990] [ 1026] 0 1026 4692 40 14 0 0 atd [56680.634995] [ 1054] 0 1054 5747 69 16 0 0 cron [56680.634999] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon [56680.635004] [ 1104] 106 1104 9630 147 23 0 0 ntpd [56680.635008] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send [56680.635013] [ 1130] 0 1130 18989 400 35 0 0 searchd [56680.635018] [ 1140] 0 1140 12307 138 28 0 -1000 sshd [56680.635022] [ 1152] 105 1152 10265 1099 24 0 0 unbound [56680.635027] [ 1172] 0 1172 1496 25 9 0 0 getty [56680.635144] [ 1173] 0 1173 1496 24 8 0 0 getty [56680.635150] [ 1031] 33 1031 87879 4814 161 0 0 apache2 [56680.635155] [ 1579] 33 1579 87941 5548 159 0 0 apache2 [56680.635160] [ 2901] 33 2901 87622 4692 158 0 0 apache2 [56680.635164] [ 4559] 33 4559 87714 4947 160 0 0 apache2 [56680.635169] [ 4561] 33 4561 87859 3852 159 0 0 apache2 [56680.635174] [ 6205] 33 6205 87294 3684 155 0 0 apache2 [56680.635179] [ 6991] 33 6991 87434 3395 152 0 0 apache2 [56680.635183] [ 6992] 33 6992 87593 3185 148 0 0 apache2 [56680.635188] [ 7289] 0 7289 10460 89 25 0 0 cron [56680.635192] [ 7297] 0 7297 1002 23 7 0 0 sh [56680.635197] [ 7303] 0 7303 7322 110 20 0 0 wget [56680.635202] [ 7363] 0 7363 10460 89 25 0 0 cron [56680.635206] [ 7374] 0 7374 1002 25 7 0 0 sh [56680.635213] [ 7378] 0 7378 7322 108 19 0 0 wget [56680.635218] [ 7437] 0 7437 10460 89 25 0 0 cron [56680.635222] [ 7438] 0 7438 10460 89 25 0 0 cron [56680.635227] [ 7439] 0 7439 10460 89 25 0 0 cron [56680.635231] [ 7443] 0 7443 1002 23 8 0 0 sh [56680.635235] [ 7444] 0 7444 1002 23 8 0 0 sh [56680.635239] [ 7445] 1337 7445 1002 24 7 0 0 sh [56680.635244] [ 7446] 1337 7446 1481 22 8 0 0 sleep [56680.635248] [ 7447] 0 7447 7322 109 19 0 0 wget [56680.635252] [ 7448] 0 7448 7322 107 19 0 0 wget [56680.635257] Out of memory: Kill process 1579 (apache2) score 11 or sacrifice child [56680.635262] Killed process 1579 (apache2) total-vm:351764kB, anon-rss:12092kB, file-rss:10100kB [56734.778939] wget invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [56734.778957] CPU: 0 PID: 7524 Comm: wget Not tainted 2.6.50-dae-xen #2 [56734.778963] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff88006d133bb0 [56734.778973] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff8800634686a0 [56734.778982] ffffffff814d3d48 0000000000000000 00000000000201da 0000000000000009 [56734.779011] Call Trace: [56734.779020] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193 [56734.779026] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8 [56734.779031] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13 [56734.779037] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca [56734.779042] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d [56734.779046] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666 [56734.779051] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f [56734.779055] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d [56734.779060] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e [56734.779065] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73 [56734.779069] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed [56734.779073] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2 [56734.779079] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324 [56734.779085] [<ffffffff81113891>] ? fsnotify+0x261/0x291 [56734.779089] [<ffffffff810ed49a>] ? vfs_fstatat+0x3e/0x8d [56734.779093] [<ffffffff810ed50b>] ? SYSC_newstat+0x10/0x27 [56734.779097] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30 [56734.779100] Mem-Info: [56734.779102] Node 0 DMA per-cpu: [56734.779106] CPU 0: hi: 0, btch: 1 usd: 0 [56734.779108] Node 0 DMA32 per-cpu: [56734.779112] CPU 0: hi: 186, btch: 31 usd: 62 [56734.779117] active_anon:18885 inactive_anon:1033 isolated_anon:0 [56734.779117] active_file:190 inactive_file:1109 isolated_file:0 [56734.779117] unevictable:0 dirty:2 writeback:0 unstable:0 [56734.779117] free:3395 slab_reclaimable:103429 slab_unreclaimable:233857 [56734.779117] mapped:3186 shmem:8327 pagetables:2038 bounce:0 [56734.779117] free_cma:0 [56734.779234] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1664kB slab_unreclaimable:4156kB kernel_stack:2056kB pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [56734.779256] lowmem_reserve[]: 0 1973 1973 1973 [56734.779267] Node 0 DMA32 free:5652kB min:5660kB low:7072kB high:8488kB active_anon:75512kB inactive_anon:4132kB active_file:760kB inactive_file:4424kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:8kB writeback:0kB mapped:12744kB shmem:33308kB slab_reclaimable:412052kB slab_unreclaimable:931272kB kernel_stack:577184kB pagetables:8148kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3285 all_unreclaimable? yes [56734.779293] lowmem_reserve[]: 0 0 0 0 [56734.779300] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 8*32kB (M) 23*64kB (EM) 8*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB [56734.779327] Node 0 DMA32: 99*4kB (E) 521*8kB (EM) 16*16kB (ER) 6*32kB (UMR) 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5652kB [56734.779351] 9645 total pagecache pages [56734.779359] 0 pages in swap cache [56734.779363] Swap cache stats: add 0, delete 0, find 0/0 [56734.779367] Free swap = 0kB [56734.779370] Total swap = 0kB [56734.785510] 524287 pages RAM [56734.785519] 12090 pages reserved [56734.785523] 286620 pages shared [56734.785526] 431108 pages non-shared [56734.785529] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [56734.785544] [ 330] 0 330 4213 114 12 0 -1000 udevd [56734.785553] [ 815] 0 815 13018 138 20 0 0 rsyslogd [56734.785560] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd [56734.785567] [ 909] 0 909 4212 113 12 0 -1000 udevd [56734.785573] [ 910] 0 910 4212 113 12 0 -1000 udevd [56734.785579] [ 911] 104 911 12487 208 27 0 0 zabbix_agentd [56734.785586] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd [56734.785593] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd [56734.785600] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd [56734.785607] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd [56734.785614] [ 1016] 0 1016 86937 2024 155 0 0 apache2 [56734.785620] [ 1026] 0 1026 4692 40 14 0 0 atd [56734.785627] [ 1054] 0 1054 5747 70 16 0 0 cron [56734.785634] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon [56734.785641] [ 1104] 106 1104 9630 147 23 0 0 ntpd [56734.785648] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send [56734.785654] [ 1130] 0 1130 18989 395 35 0 0 searchd [56734.785661] [ 1140] 0 1140 12307 138 28 0 -1000 sshd [56734.785665] [ 1152] 105 1152 10265 1099 24 0 0 unbound [56734.785670] [ 1172] 0 1172 1496 25 9 0 0 getty [56734.785712] [ 1173] 0 1173 1496 24 8 0 0 getty [56734.785718] [ 1031] 33 1031 87879 4814 161 0 0 apache2 [56734.785722] [ 2901] 33 2901 87622 4692 158 0 0 apache2 [56734.785727] [ 4559] 33 4559 87714 4954 160 0 0 apache2 [56734.785731] [ 4561] 33 4561 87859 3815 159 0 0 apache2 [56734.785737] [ 6205] 33 6205 87294 3674 155 0 0 apache2 [56734.785741] [ 6991] 33 6991 87434 3398 152 0 0 apache2 [56734.785745] [ 6992] 33 6992 87593 3228 150 0 0 apache2 [56734.785750] [ 7363] 0 7363 10460 89 25 0 0 cron [56734.785754] [ 7374] 0 7374 1002 25 7 0 0 sh [56734.785760] [ 7378] 0 7378 7322 108 19 0 0 wget [56734.785765] [ 7505] 0 7505 10460 89 25 0 0 cron [56734.785769] [ 7506] 0 7506 10460 89 25 0 0 cron [56734.785773] [ 7507] 0 7507 10460 89 25 0 0 cron [56734.785777] [ 7508] 0 7508 10460 89 25 0 0 cron [56734.785782] [ 7509] 0 7509 10460 89 25 0 0 cron [56734.785786] [ 7510] 0 7510 10460 89 25 0 0 cron [56734.785790] [ 7517] 0 7517 1002 24 8 0 0 sh [56734.785794] [ 7518] 0 7518 1002 24 8 0 0 sh [56734.785800] [ 7519] 0 7519 1002 24 8 0 0 sh [56734.785804] [ 7520] 0 7520 1002 23 8 0 0 sh [56734.785808] [ 7521] 1337 7521 1002 24 8 0 0 sh [56734.785813] [ 7522] 0 7522 1002 23 7 0 0 sh [56734.785817] [ 7523] 1337 7523 1481 21 8 0 0 sleep [56734.785822] [ 7524] 0 7524 7322 148 20 0 0 wget [56734.785826] [ 7525] 0 7525 7322 111 20 0 0 wget [56734.785830] [ 7526] 0 7526 7322 110 19 0 0 wget [56734.785834] [ 7527] 0 7527 7322 110 20 0 0 wget [56734.785839] [ 7528] 0 7528 1002 24 7 0 0 maxlifetime [56734.785846] [ 7530] 0 7530 16356 291 37 0 0 php5 [56734.785850] Out of memory: Kill process 4559 (apache2) score 9 or sacrifice child [56734.785854] Killed process 4559 (apache2) total-vm:350856kB, anon-rss:11596kB, file-rss:8220kB [56793.525886] wget invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [56793.525921] CPU: 0 PID: 7614 Comm: wget Not tainted 2.6.50-dae-xen #2 [56793.525928] 0000000000000000 ffffffff8134ab12 ffffffff8134f478 00000000000007c5 [56793.525938] 0000000000000010 ffffffff81089ad6 ffffffff8134f09a ffff880001a127c0 [56793.525946] ffffffff814d3d48 0000000000000000 00000000000201da 0000000000000009 [56793.525976] Call Trace: [56793.525986] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193 [56793.526012] [<ffffffff8134f478>] ? retint_restore_args+0x5/0x6 [56793.526042] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8 [56793.526048] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13 [56793.526057] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca [56793.526064] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d [56793.526070] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666 [56793.526082] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f [56793.526090] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d [56793.526097] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e [56793.526104] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73 [56793.526111] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed [56793.526117] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2 [56793.526125] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324 [56793.526133] [<ffffffff81113891>] ? fsnotify+0x261/0x291 [56793.526142] [<ffffffff810ed49a>] ? vfs_fstatat+0x3e/0x8d [56793.526149] [<ffffffff810ed50b>] ? SYSC_newstat+0x10/0x27 [56793.526155] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30 [56793.526160] Mem-Info: [56793.526163] Node 0 DMA per-cpu: [56793.526171] CPU 0: hi: 0, btch: 1 usd: 0 [56793.526175] Node 0 DMA32 per-cpu: [56793.526180] CPU 0: hi: 186, btch: 31 usd: 150 [56793.526188] active_anon:17848 inactive_anon:1033 isolated_anon:5 [56793.526188] active_file:78 inactive_file:778 isolated_file:0 [56793.526188] unevictable:0 dirty:0 writeback:0 unstable:0 [56793.526188] free:3395 slab_reclaimable:104385 slab_unreclaimable:234096 [56793.526188] mapped:2974 shmem:8327 pagetables:2047 bounce:0 [56793.526188] free_cma:0 [56793.526213] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:28kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1664kB slab_unreclaimable:4156kB kernel_stack:2056kB pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3 all_unreclaimable? yes [56793.526238] lowmem_reserve[]: 0 1973 1973 1973 [56793.526247] Node 0 DMA32 free:5652kB min:5660kB low:7072kB high:8488kB active_anon:71364kB inactive_anon:4132kB active_file:312kB inactive_file:3100kB unevictable:0kB isolated(anon):20kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:11896kB shmem:33308kB slab_reclaimable:415876kB slab_unreclaimable:932228kB kernel_stack:577872kB pagetables:8184kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1295 all_unreclaimable? yes [56793.526277] lowmem_reserve[]: 0 0 0 0 [56793.526284] Node 0 DMA: 68*4kB (UEM) 37*8kB (UEM) 16*16kB (UEM) 8*32kB (M) 23*64kB (EM) 8*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB [56793.526316] Node 0 DMA32: 3*4kB (UEM) 621*8kB (UEM) 2*16kB (UR) 0*32kB 0*64kB 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5652kB [56793.526343] 9197 total pagecache pages [56793.526347] 0 pages in swap cache [56793.526352] Swap cache stats: add 0, delete 0, find 0/0 [56793.526356] Free swap = 0kB [56793.526359] Total swap = 0kB [56793.532348] 524287 pages RAM [56793.532355] 12090 pages reserved [56793.532358] 284892 pages shared [56793.532360] 430988 pages non-shared [56793.532363] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [56793.532376] [ 330] 0 330 4213 114 12 0 -1000 udevd [56793.532381] [ 815] 0 815 13018 138 20 0 0 rsyslogd [56793.532386] [ 853] 104 853 12487 102 27 0 0 zabbix_agentd [56793.532391] [ 909] 0 909 4212 113 12 0 -1000 udevd [56793.532396] [ 910] 0 910 4212 113 12 0 -1000 udevd [56793.532400] [ 911] 104 911 12487 207 27 0 0 zabbix_agentd [56793.532405] [ 912] 104 912 12487 101 27 0 0 zabbix_agentd [56793.532409] [ 913] 104 913 12487 101 27 0 0 zabbix_agentd [56793.532414] [ 914] 104 914 12487 101 27 0 0 zabbix_agentd [56793.532418] [ 915] 104 915 13005 109 29 0 0 zabbix_agentd [56793.532424] [ 1016] 0 1016 86937 2028 155 0 0 apache2 [56793.532428] [ 1026] 0 1026 4692 40 14 0 0 atd [56793.532433] [ 1054] 0 1054 5747 70 16 0 0 cron [56793.532437] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon [56793.532442] [ 1104] 106 1104 9630 147 23 0 0 ntpd [56793.532447] [ 1117] 8 1117 2947 45 12 0 0 nullmailer-send [56793.532451] [ 1130] 0 1130 18989 374 35 0 0 searchd [56793.532456] [ 1140] 0 1140 12307 138 28 0 -1000 sshd [56793.532461] [ 1152] 105 1152 10265 1099 24 0 0 unbound [56793.532467] [ 1172] 0 1172 1496 25 9 0 0 getty [56793.532484] [ 1173] 0 1173 1496 24 8 0 0 getty [56793.532491] [ 1031] 33 1031 87879 4791 161 0 0 apache2 [56793.532497] [ 2901] 33 2901 87622 4723 158 0 0 apache2 [56793.532504] [ 4561] 33 4561 87859 3866 159 0 0 apache2 [56793.532510] [ 6205] 33 6205 87294 3663 155 0 0 apache2 [56793.532515] [ 6991] 33 6991 87434 3398 152 0 0 apache2 [56793.532522] [ 6992] 33 6992 87593 3228 150 0 0 apache2 [56793.532526] [ 7363] 0 7363 10460 89 25 0 0 cron [56793.532531] [ 7374] 0 7374 1002 25 7 0 0 sh [56793.532537] [ 7378] 0 7378 7322 108 19 0 0 wget [56793.532547] [ 7555] 33 7555 86937 1967 137 0 0 apache2 [56793.532553] [ 7593] 0 7593 10460 88 25 0 0 cron [56793.532559] [ 7594] 0 7594 10460 88 25 0 0 cron [56793.532564] [ 7595] 0 7595 10460 88 25 0 0 cron [56793.532568] [ 7596] 0 7596 10460 88 25 0 0 cron [56793.532573] [ 7597] 0 7597 10460 88 25 0 0 cron [56793.532580] [ 7598] 0 7598 10460 88 25 0 0 cron [56793.532584] [ 7599] 0 7599 10460 88 25 0 0 cron [56793.532588] [ 7607] 0 7607 1002 25 8 0 0 sh [56793.532593] [ 7608] 1337 7608 1002 24 8 0 0 sh [56793.532597] [ 7609] 0 7609 1002 26 7 0 0 sh [56793.532601] [ 7610] 0 7610 1002 24 7 0 0 sh [56793.532606] [ 7611] 0 7611 1002 25 7 0 0 sh [56793.532611] [ 7612] 0 7612 1002 23 8 0 0 sh [56793.532615] [ 7613] 0 7613 1002 25 7 0 0 sh [56793.532620] [ 7614] 0 7614 7322 156 20 0 0 wget [56793.532624] [ 7615] 1337 7615 1481 19 8 0 0 sleep [56793.532628] [ 7616] 0 7616 7322 110 19 0 0 wget [56793.532632] [ 7617] 0 7617 7322 110 20 0 0 wget [56793.532637] [ 7618] 0 7618 7322 109 19 0 0 wget [56793.532641] [ 7619] 0 7619 7322 109 20 0 0 wget [56793.532646] [ 7620] 0 7620 7322 109 20 0 0 wget [56793.532652] Out of memory: Kill process 1031 (apache2) score 9 or sacrifice child [56793.532656] Killed process 1031 (apache2) total-vm:351516kB, anon-rss:9624kB, file-rss:9540kB ... [57131.763804] zabbix_sender invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [57131.763840] CPU: 0 PID: 8158 Comm: zabbix_sender Not tainted 2.6.50-dae-xen #2 [57131.763847] 0000000000000000 ffffffff8134ab12 0000000000000080 ffff880073e21bb0 [57131.763858] 0000000000000000 ffffffff81089ad6 ffffffff8134f09a ffff8800788aea00 [57131.763867] ffffffff814d3d48 0000000000000000 00000000000201da 0000000000000004 [57131.763885] Call Trace: [57131.763896] [<ffffffff8134ab12>] ? dump_header.isra.9+0x69/0x193 [57131.763906] [<ffffffff81089ad6>] ? arch_local_irq_restore+0x7/0x8 [57131.763913] [<ffffffff8134f09a>] ? _raw_spin_unlock_irqrestore+0x11/0x13 [57131.763921] [<ffffffff810b0d0f>] ? oom_kill_process+0x6a/0x2ca [57131.763947] [<ffffffff810b13da>] ? out_of_memory+0x324/0x33d [57131.763955] [<ffffffff810b4e45>] ? __alloc_pages_nodemask+0x55c/0x666 [57131.763962] [<ffffffff810b0394>] ? filemap_fault+0x1fe/0x35f [57131.763969] [<ffffffff810c5758>] ? __do_fault+0xa7/0x33d [57131.763977] [<ffffffff81005de7>] ? get_phys_to_machine+0x12/0x5e [57131.763985] [<ffffffff81003ca7>] ? mfn_to_pfn.part.32+0x50/0x73 [57131.763991] [<ffffffff810c7b21>] ? handle_pte_fault+0x13f/0x6ed [57131.763997] [<ffffffff810c8fbd>] ? handle_mm_fault+0x1a2/0x1f2 [57131.764006] [<ffffffff81025fec>] ? __do_page_fault+0x1ac/0x324 [57131.764017] [<ffffffff810cbcbc>] ? __split_vma.isra.25+0x11a/0x182 [57131.764023] [<ffffffff810ce298>] ? mprotect_fixup+0x1bd/0x1f7 [57131.764030] [<ffffffff8134f6e5>] ? page_fault+0x25/0x30 [57131.764034] Mem-Info: [57131.764037] Node 0 DMA per-cpu: [57131.764041] CPU 0: hi: 0, btch: 1 usd: 0 [57131.764045] Node 0 DMA32 per-cpu: [57131.764049] CPU 0: hi: 186, btch: 31 usd: 62 [57131.764076] active_anon:11743 inactive_anon:1026 isolated_anon:0 [57131.764076] active_file:288 inactive_file:1121 isolated_file:0 [57131.764076] unevictable:0 dirty:0 writeback:0 unstable:0 [57131.764076] free:3369 slab_reclaimable:109088 slab_unreclaimable:235420 [57131.764076] mapped:358 shmem:8328 pagetables:821 bounce:0 [57131.764076] free_cma:0 [57131.764092] Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1760kB slab_unreclaimable:4152kB kernel_stack:2056kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [57131.764115] lowmem_reserve[]: 0 1973 1973 1973 [57131.764122] Node 0 DMA32 free:5548kB min:5660kB low:7072kB high:8488kB active_anon:46972kB inactive_anon:4104kB active_file:1152kB inactive_file:4472kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080768kB managed:2020992kB mlocked:0kB dirty:0kB writeback:0kB mapped:1432kB shmem:33312kB slab_reclaimable:434592kB slab_unreclaimable:937528kB kernel_stack:582224kB pagetables:3284kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4240 all_unreclaimable? yes [57131.764151] lowmem_reserve[]: 0 0 0 0 [57131.764158] Node 0 DMA: 64*4kB (UE) 33*8kB (UEM) 13*16kB (UEM) 11*32kB (UEM) 19*64kB (EM) 8*128kB (M) 0*256kB 1*512kB (M) 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB [57131.764189] Node 0 DMA32: 383*4kB (UEM) 164*8kB (UEM) 79*16kB (UMR) 17*32kB (UEMR) 4*64kB (EM) 1*128kB (R) 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 5548kB [57131.764242] 9744 total pagecache pages [57131.764253] 0 pages in swap cache [57131.764256] Swap cache stats: add 0, delete 0, find 0/0 [57131.764263] Free swap = 0kB [57131.764267] Total swap = 0kB [57131.770252] 524287 pages RAM [57131.770260] 12090 pages reserved [57131.770262] 265440 pages shared [57131.770265] 433233 pages non-shared [57131.770267] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [57131.770280] [ 330] 0 330 4213 114 12 0 -1000 udevd [57131.770285] [ 815] 0 815 13048 150 20 0 0 rsyslogd [57131.770290] [ 853] 104 853 12486 99 27 0 0 zabbix_agentd [57131.770295] [ 909] 0 909 4212 113 12 0 -1000 udevd [57131.770299] [ 910] 0 910 4212 113 12 0 -1000 udevd [57131.770305] [ 1016] 0 1016 86937 2031 155 0 0 apache2 [57131.770310] [ 1054] 0 1054 5747 69 16 0 0 cron [57131.770314] [ 1076] 103 1076 5830 54 15 0 0 dbus-daemon [57131.770319] [ 1104] 106 1104 9630 147 23 0 0 ntpd [57131.770323] [ 1130] 0 1130 18989 374 35 0 0 searchd [57131.770328] [ 1140] 0 1140 12307 138 28 0 -1000 sshd [57131.770334] [ 7363] 0 7363 10460 89 25 0 0 cron [57131.770339] [ 7374] 0 7374 1002 25 7 0 0 sh [57131.770343] [ 7378] 0 7378 7322 108 19 0 0 wget [57131.770348] [ 8003] 0 8003 10460 88 25 0 0 cron [57131.770352] [ 8006] 0 8006 10460 88 25 0 0 cron [57131.770357] [ 8009] 0 8009 10460 88 25 0 0 cron [57131.770361] [ 8020] 1337 8020 1002 23 7 0 0 sh [57131.770366] [ 8023] 0 8023 1002 23 6 0 0 sh [57131.770370] [ 8024] 0 8024 1002 24 7 0 0 sh [57131.770399] [ 8030] 0 8030 7322 108 19 0 0 wget [57131.770406] [ 8035] 0 8035 7322 109 20 0 0 wget [57131.770410] [ 8051] 0 8051 1002 41 8 0 0 rc [57131.770414] [ 8054] 0 8054 3085 60 11 0 0 startpar [57131.770419] [ 8055] 1337 8055 1002 23 8 0 0 zabbix-launch-t [57131.770424] [ 8057] 0 8057 1002 40 7 0 0 apache2 [57131.770430] [ 8067] 1337 8067 2841 76 9 0 0 apache [57131.770436] [ 8084] 0 8084 1002 24 8 0 0 apache2ctl [57131.770443] [ 8092] 0 8092 26561 288 48 0 0 apache2 [57131.770450] [ 8115] 0 8115 1002 22 8 0 0 zabbix-agent [57131.770457] [ 8116] 0 8116 980 32 8 0 0 start-stop-daem [57131.770461] [ 8158] 1337 8158 10130 168 22 0 0 zabbix_sender [57131.770466] [ 8161] 33 8161 86937 1967 137 0 0 apache2 [57131.770471] Out of memory: Kill process 8161 (apache2) score 4 or sacrifice child [57131.770475] Killed process 8161 (apache2) total-vm:347748kB, anon-rss:7868kB, file-rss:0kB _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |