[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Kernel 3.11 / 3.12 OOM killer and Xen ballooning



Bob Liu wrote:
On 01/07/2014 05:21 PM, James Dingwall wrote:
Bob Liu wrote:
Could you confirm that this problem doesn't exist if loading tmem with
selfshrinking=0 during compile gcc? It seems that you are compiling
difference packages during your testing.
This will help to figure out whether selfshrinking is the root cause.
Got an oom with selfshrinking=0, again during a gcc compile.
Unfortunately I don't have a single test case which demonstrates the
problem but as I mentioned before it will generally show up under
compiles of large packages such as glibc, kdelibs, gcc etc.

So the root cause is not because enabled selfshrinking.
Then what I can think of is that the xen-selfballoon driver was too
aggressive, too many pages were ballooned out which causeed heavy memory
pressure to guest OS.
And kswapd started to reclaim page until most of pages were
unreclaimable(all_unreclaimable=yes for all zones), then OOM Killer was
triggered.
In theory the balloon driver should give back ballooned out pages to
guest OS, but I'm afraid this procedure is not fast enough.

My suggestion is reserve a min memory for your guest OS so that the
xen-selfballoon won't be so aggressive.
You can do it through parameters selfballoon_reserved_mb or
selfballoon_min_usable_mb.

I don't know if this is a separate or related issue but over the
holidays I also had a problem with six of the guests on my system where
kswapd was running at 100% and had clocked up >9000 minutes of cpu time
even though there was otherwise no load on them.  Of the guests I
restarted yesterday in this state two have already got in to the same
state again, they are running a kernel with the first patch that you sent.
As soon as I echo 32 both (originally 0)
/sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_reserved_mb
/sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_min_usable_mb
Then the kswapd process stopped running at 100%. Unfortunately I didn't check between the two commands to see if one by itself made a difference but I'll look for that next time.
Could you get the meminfo in guest OS at that time?
After
cat /proc/meminfo
MemTotal:         397028 kB
MemFree:          163756 kB
Buffers:            1260 kB
Cached:           129284 kB
SwapCached:          132 kB
Active:            22664 kB
Inactive:         159576 kB
Active(anon):       8004 kB
Inactive(anon):    44412 kB
Active(file):      14660 kB
Inactive(file):   115164 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       2097148 kB
SwapFree:        2096896 kB
Dirty:                20 kB
Writeback:             0 kB
AnonPages:         51640 kB
Mapped:            14136 kB
Shmem:               720 kB
Slab:              19492 kB
SReclaimable:       7692 kB
SUnreclaim:        11800 kB
KernelStack:        1816 kB
PageTables:         7928 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     2295660 kB
Committed_AS:     338552 kB
VmallocTotal:   34359738367 kB
VmallocUsed:        9020 kB
VmallocChunk:   34359716408 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
DirectMap4k:     1048576 kB
DirectMap2M:           0 kB

cat /proc/vmstat
nr_free_pages 40916
nr_alloc_batch 0
nr_inactive_anon 11102
nr_active_anon 2009
nr_inactive_file 28791
nr_active_file 3665
nr_unevictable 0
nr_mlock 0
nr_anon_pages 12904
nr_mapped 3534
nr_file_pages 32669
nr_dirty 5
nr_writeback 0
nr_slab_reclaimable 1923
nr_slab_unreclaimable 2945
nr_page_table_pages 1982
nr_kernel_stack 227
nr_unstable 0
nr_bounce 0
nr_vmscan_write 781891
nr_vmscan_immediate_reclaim 6245
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 180
nr_dirtied 86609
nr_written 861010
numa_hit 8353372
numa_miss 0
numa_foreign 0
numa_interleave 0
numa_local 8353372
numa_other 0
nr_anon_transparent_hugepages 0
nr_free_cma 0
nr_dirty_threshold 16991
nr_dirty_background_threshold 8495
pgpgin 2044575
pgpgout 645866
pswpin 123
pswpout 153
pgalloc_dma 164944
pgalloc_dma32 7347917
pgalloc_normal 1032559
pgalloc_movable 0
pgfree 8586607
pgactivate 2012718
pgdeactivate 2276721
pgfault 7295414
pgmajfault 345301
pgrefill_dma 55271
pgrefill_dma32 2263007
pgrefill_normal 1771
pgrefill_movable 0
pgsteal_kswapd_dma 44880
pgsteal_kswapd_dma32 2587500
pgsteal_kswapd_normal 0
pgsteal_kswapd_movable 0
pgsteal_direct_dma 0
pgsteal_direct_dma32 37
pgsteal_direct_normal 0
pgsteal_direct_movable 0
pgscan_kswapd_dma 204749
pgscan_kswapd_dma32 4477230
pgscan_kswapd_normal 0
pgscan_kswapd_movable 0
pgscan_direct_dma 0
pgscan_direct_dma32 39
pgscan_direct_normal 0
pgscan_direct_movable 0
pgscan_direct_throttle 0
zone_reclaim_failed 0
pginodesteal 0
slabs_scanned 2720128
kswapd_inodesteal 41065
kswapd_low_wmark_hit_quickly 14897
kswapd_high_wmark_hit_quickly 116697740
pageoutrun 116717997
allocstall 1
pgrotated 8497
numa_pte_updates 0
numa_huge_pte_updates 0
numa_hint_faults 0
numa_hint_faults_local 0
numa_pages_migrated 0
pgmigrate_success 0
pgmigrate_fail 0
compact_migrate_scanned 0
compact_free_scanned 0
compact_isolated 0
compact_stall 0
compact_fail 0
compact_success 0
unevictable_pgs_culled 29365
unevictable_pgs_scanned 0
unevictable_pgs_rescued 29145
unevictable_pgs_mlocked 29550
unevictable_pgs_munlocked 29550
unevictable_pgs_cleared 0
unevictable_pgs_stranded 0
thp_fault_alloc 0
thp_fault_fallback 0
thp_collapse_alloc 0
thp_collapse_alloc_failed 0
thp_split 0
thp_zero_page_alloc 0
thp_zero_page_alloc_failed 0
nr_tlb_remote_flush 10780
nr_tlb_remote_flush_received 21564
nr_tlb_local_flush_all 66247
nr_tlb_local_flush_one 1446496


Thanks,
-Bob

/sys/module/tmem/parameters/cleancache Y
/sys/module/tmem/parameters/frontswap Y
/sys/module/tmem/parameters/selfballooning Y
/sys/module/tmem/parameters/selfshrinking N

James

[ 8212.940520] cc1plus invoked oom-killer: gfp_mask=0x200da, order=0,
oom_score_adj=0
[ 8212.940529] CPU: 1 PID: 23678 Comm: cc1plus Tainted: G W    3.12.5 #88
[ 8212.940532]  ffff88001e38cdf8 ffff88000094f968 ffffffff8148f200
ffff88001f90e8e8
[ 8212.940536]  ffff88001e38c8c0 ffff88000094fa08 ffffffff8148ccf7
ffff88000094f9b8
[ 8212.940538]  ffffffff810f8d97 ffff88000094f998 ffffffff81006dc8
ffff88000094f9a8
[ 8212.940542] Call Trace:
[ 8212.940554]  [<ffffffff8148f200>] dump_stack+0x46/0x58
[ 8212.940558]  [<ffffffff8148ccf7>] dump_header.isra.9+0x6d/0x1cc
[ 8212.940564]  [<ffffffff810f8d97>] ? super_cache_count+0xa8/0xb8
[ 8212.940569]  [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
[ 8212.940573]  [<ffffffff81006ea9>] ? xen_clocksource_get_cycles+0x9/0xb
[ 8212.940578]  [<ffffffff81494abe>] ?
_raw_spin_unlock_irqrestore+0x47/0x62
[ 8212.940583]  [<ffffffff81296b27>] ? ___ratelimit+0xcb/0xe8
[ 8212.940588]  [<ffffffff810b2bbf>] oom_kill_process+0x70/0x2fd
[ 8212.940592]  [<ffffffff810bca0e>] ? zone_reclaimable+0x11/0x1e
[ 8212.940597]  [<ffffffff81048779>] ? has_ns_capability_noaudit+0x12/0x19
[ 8212.940600]  [<ffffffff81048792>] ? has_capability_noaudit+0x12/0x14
[ 8212.940603]  [<ffffffff810b32de>] out_of_memory+0x31b/0x34e
[ 8212.940608]  [<ffffffff810b7438>] __alloc_pages_nodemask+0x65b/0x792
[ 8212.940612]  [<ffffffff810e3da3>] alloc_pages_vma+0xd0/0x10c
[ 8212.940617]  [<ffffffff810dd5a4>] read_swap_cache_async+0x70/0x120
[ 8212.940620]  [<ffffffff810dd6e4>] swapin_readahead+0x90/0xd4
[ 8212.940623]  [<ffffffff81005b35>] ? pte_mfn_to_pfn+0x59/0xcb
[ 8212.940627]  [<ffffffff810cf99d>] handle_mm_fault+0x8a4/0xd54
[ 8212.940630]  [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
[ 8212.940634]  [<ffffffff810115d2>] ? sched_clock+0x9/0xd
[ 8212.940638]  [<ffffffff8106772f>] ? sched_clock_local+0x12/0x75
[ 8212.940641]  [<ffffffff8106823b>] ? arch_vtime_task_switch+0x81/0x86
[ 8212.940646]  [<ffffffff81037f40>] __do_page_fault+0x3d8/0x437
[ 8212.940649]  [<ffffffff81006dc8>] ? xen_clocksource_read+0x20/0x22
[ 8212.940652]  [<ffffffff810115d2>] ? sched_clock+0x9/0xd
[ 8212.940654]  [<ffffffff8106772f>] ? sched_clock_local+0x12/0x75
[ 8212.940658]  [<ffffffff810a45cc>] ? __acct_update_integrals+0xb4/0xbf
[ 8212.940661]  [<ffffffff810a493f>] ? acct_account_cputime+0x17/0x19
[ 8212.940663]  [<ffffffff81067c28>] ? account_user_time+0x67/0x92
[ 8212.940666]  [<ffffffff8106811b>] ? vtime_account_user+0x4d/0x52
[ 8212.940669]  [<ffffffff81037fd8>] do_page_fault+0x1a/0x5a
[ 8212.940674]  [<ffffffff810a065f>] ? rcu_user_enter+0xe/0x10
[ 8212.940677]  [<ffffffff81495158>] page_fault+0x28/0x30
[ 8212.940679] Mem-Info:
[ 8212.940681] Node 0 DMA per-cpu:
[ 8212.940684] CPU    0: hi:    0, btch:   1 usd:   0
[ 8212.940685] CPU    1: hi:    0, btch:   1 usd:   0
[ 8212.940686] Node 0 DMA32 per-cpu:
[ 8212.940688] CPU    0: hi:  186, btch:  31 usd: 116
[ 8212.940690] CPU    1: hi:  186, btch:  31 usd: 124
[ 8212.940691] Node 0 Normal per-cpu:
[ 8212.940693] CPU    0: hi:    0, btch:   1 usd:   0
[ 8212.940694] CPU    1: hi:    0, btch:   1 usd:   0
[ 8212.940700] active_anon:105765 inactive_anon:105882 isolated_anon:0
  active_file:8412 inactive_file:8612 isolated_file:0
  unevictable:0 dirty:0 writeback:0 unstable:0
  free:1143 slab_reclaimable:3575 slab_unreclaimable:3464
  mapped:3792 shmem:6 pagetables:2534 bounce:0
  free_cma:0 totalram:246132 balloontarget:306242
[ 8212.940702] Node 0 DMA free:1964kB min:88kB low:108kB high:132kB
active_anon:5092kB inactive_anon:5328kB active_file:416kB
inactive_file:608kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:15996kB managed:15392kB mlocked:0kB dirty:0kB
writeback:0kB mapped:320kB shmem:0kB slab_reclaimable:252kB
slab_unreclaimable:492kB kernel_stack:120kB pagetables:252kB
unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:26951 all_unreclaimable? yes
[ 8212.940711] lowmem_reserve[]: 0 469 469 469
[ 8212.940715] Node 0 DMA32 free:2608kB min:2728kB low:3408kB
high:4092kB active_anon:181456kB inactive_anon:181528kB
active_file:22296kB inactive_file:22644kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:507904kB managed:466364kB
mlocked:0kB dirty:0kB writeback:0kB mapped:8628kB shmem:20kB
slab_reclaimable:10756kB slab_unreclaimable:12548kB kernel_stack:1688kB
pagetables:8876kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:612393 all_unreclaimable? yes
[ 8212.940722] lowmem_reserve[]: 0 0 0 0
[ 8212.940725] Node 0 Normal free:0kB min:0kB low:0kB high:0kB
active_anon:236512kB inactive_anon:236672kB active_file:10936kB
inactive_file:11196kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:524288kB managed:502772kB mlocked:0kB
dirty:0kB writeback:0kB mapped:6220kB shmem:4kB slab_reclaimable:3292kB
slab_unreclaimable:816kB kernel_stack:64kB pagetables:1008kB
unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:745963 all_unreclaimable? yes
[ 8212.940732] lowmem_reserve[]: 0 0 0 0
[ 8212.940735] Node 0 DMA: 1*4kB (R) 0*8kB 4*16kB (R) 1*32kB (R) 1*64kB
(R) 2*128kB (R) 0*256kB 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 1956kB
[ 8212.940747] Node 0 DMA32: 652*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2608kB
[ 8212.940756] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[ 8212.940765] 16847 total pagecache pages
[ 8212.940766] 8381 pages in swap cache
[ 8212.940768] Swap cache stats: add 741397, delete 733016, find
250268/342284
[ 8212.940769] Free swap  = 1925576kB
[ 8212.940770] Total swap = 2097148kB
[ 8212.951044] 262143 pages RAM
[ 8212.951046] 11939 pages reserved
[ 8212.951047] 540820 pages shared
[ 8212.951048] 240248 pages non-shared
[ 8212.951050] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents
oom_score_adj name
<snip process list>
[ 8212.951310] Out of memory: Kill process 23721 (cc1plus) score 119 or
sacrifice child
[ 8212.951313] Killed process 23721 (cc1plus) total-vm:530268kB,
anon-rss:350980kB, file-rss:9408kB
[54810.683658] kjournald starting.  Commit interval 5 seconds
[54810.684381] EXT3-fs (xvda1): using internal journal
[54810.684402] EXT3-fs (xvda1): mounted filesystem with writeback data mode



--

*James Dingwall*

Script Monkey

zynstra-signature-logo <http://www.zynstra.com/>twitter-black <http://www.twitter.com/zynstra>linkedin-black <http://www.linkedin.com/company/zynstra>

Zynstra is a private limited company registered in England and Wales (registered number 07864369). Our registered office is 5 New Street Square, London, EC4A 3TW and our headquarters are at Bath Ventures, Broad Quay, Bath, BA1 1UD.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.