Re: [Xen-devel] Re: 2.6.32 pvops dom0 oom when creating a domU

[Pasi Kärkkäinen <pasik@xxxxxx>]

On Tue, Aug 02, 2011 at 01:57:53AM +0200, Daniel Musketa wrote:
> 2011/8/2 Daniel Musketa <daniel.musketa@xxxxxxxxxxxxxx>:
> > Maybe I should say more about memory usage:
> 
> I got even more detailled logs for you in the attached file:
> 
> line   3: dom0 memory: 2,560 M
> line  20: 21,637 M free
> line  34: dom0_mem=4000M
> line  47: domU memory = '2800'
> line  72: OOM begins
> line 157: key 'm' on (XEN) console
> 

Please try editing /etc/xen/xend-config.sxp and disable memory ballooning 
totally.
(ie. set option "dom0-min-mem" to match dom0 memory, and 
"enable-dom0-ballooning" to no).

Then try again with both dom0_mem=4096M for xen.gz and mem=4096M for Linux dom0.
Does that work better? 

Konrad was guessing that xend might be ballooning dom0 down even when there's 
free memory in xen,
and thus causing that problem.. so better make sure ballooning is disabled.

-- Pasi


> 
> Daniel

> root@2ven:/etc/xen# xm li
> Name                                        ID   Mem VCPUs      State   
> Time(s)
> Domain-0                                     0  2560    12     r-----    111.1
> joel                                         1  1500    12     -b----      5.3
> 
> 
> 
> root@2ven:/etc/xen# xm info
> host                   : 2ven
> release                : 2.6.32.43
> version                : #3 SMP Mon Jul 25 23:39:42 CEST 2011
> machine                : x86_64
> nr_cpus                : 12
> nr_nodes               : 1
> cores_per_socket       : 6
> threads_per_core       : 2
> cpu_mhz                : 2400
> hw_caps                : 
> bfebfbff:2c100800:00000000:00001f40:029ee3ff:00000000:00000001:00000000
> virt_caps              : hvm
> total_memory           : 24515
> free_memory            : 21637
> node_to_cpu            : node0:0-11
> node_to_memory         : node0:21637
> node_to_dma32_mem      : node0:2174
> max_node_id            : 0
> xen_major              : 4
> xen_minor              : 0
> xen_extra              : .2
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          : unavailable
> xen_commandline        : dom0_mem=4000M loglvl=all guest_loglvl=all 
> com1=115200,8n1 console=com1 sync_console console_to_ring lapic=debug 
> apic_verbosity=debug apic=debug iommu=off
> cc_compiler            : gcc version 4.4.5 (Debian 4.4.5-8)
> cc_compile_by          : root
> cc_compile_domain      : (none)
> cc_compile_date        : Mon Aug  1 11:16:18 CEST 2011
> xend_config_format     : 4
> 
> 
> 
> root@2ven:/etc/xen# cat clapton.cfg
> name        = 'clapton'
> kernel      = '/root/domU-boot/clapton/vmlinuz'
> ramdisk     = '/root/domU-boot/clapton/initrd.img'
> memory      = '2800'
> vcpus = 12
> cpu_weight = 512
> root        = 'LABEL=clapton_root ro'
> disk        = [
>                   'phy:/dev/zven/clapton_root,sda1,w',
>                   'phy:/dev/zven/clapton_samba,sda2,w',
>                   'phy:/dev/zven/clapton_scan,sda3,w',
>                   'phy:/dev/zven/clapton_mail,sda4,w',
>                   'phy:/dev/zven/clapton_tmp,sda6,w',
>                   'phy:/dev/zven/clapton_swap,sda7,w',
>                   'phy:/dev/mapper/zven-ISO_sage2,sdc,r',
>               ]
> vif         = [ 'mac=00:16:3E:88:88:88' ]
> extra = 'xencons=tty'
> 
> 
> 
> root@2ven:/etc/xen# xm cre clapton.cfg
> Using config file "./clapton.cfg".
> 
> 
> ==> OOM
> 
> 
> [  566.419826] rs:main Q:Reg invoked oom-killer: gfp_mask=0x201da, order=0, 
> oom_adj=0
> [  566.428172] rs:main Q:Reg cpuset=/ mems_allowed=0
> [  566.433503] Pid: 3418, comm: rs:main Q:Reg Not tainted 2.6.32.43 #3
> [  566.440582] Call Trace:
> [  566.443396]  [<ffffffff810b5605>] ? T.463+0x7c/0x239
> [  566.449015]  [<ffffffff810b58f6>] ? __out_of_memory+0x134/0x14b
> [  566.455704]  [<ffffffff810b5a49>] ? out_of_memory+0x13c/0x16e
> [  566.462201]  [<ffffffff810b9397>] ? __alloc_pages_nodemask+0x4b3/0x5d0
> [  566.469570]  [<ffffffff810bacbb>] ? __do_page_cache_readahead+0x93/0x1b3
> [  566.477132]  [<ffffffff810badf7>] ? ra_submit+0x1c/0x20
> [  566.483045]  [<ffffffff810b4b12>] ? filemap_fault+0x17e/0x300
> [  566.489542]  [<ffffffff8100ca36>] ? xen_make_pte+0x7d/0x7f
> [  566.495748]  [<ffffffff810cc26a>] ? __do_fault+0x52/0x3b3
> [  566.501855]  [<ffffffff8100c2e5>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
> [  566.509708]  [<ffffffff810ccff6>] ? handle_mm_fault+0x393/0x81b
> [  566.516400]  [<ffffffff8130b873>] ? do_page_fault+0x2f0/0x30c
> [  566.522893]  [<ffffffff813097c5>] ? page_fault+0x25/0x30
> [  566.528905] Mem-Info:
> [  566.531522] Node 0 DMA per-cpu:
> [  566.535108] CPU    0: hi:    0, btch:   1 usd:   0
> [  566.540539] CPU    1: hi:    0, btch:   1 usd:   0
> [  566.545967] CPU    2: hi:    0, btch:   1 usd:   0
> [  566.551397] CPU    3: hi:    0, btch:   1 usd:   0
> [  566.556826] CPU    4: hi:    0, btch:   1 usd:   0
> [  566.562256] CPU    5: hi:    0, btch:   1 usd:   0
> [  566.567685] CPU    6: hi:    0, btch:   1 usd:   0
> [  566.573115] CPU    7: hi:    0, btch:   1 usd:   0
> [  566.578544] CPU    8: hi:    0, btch:   1 usd:   0
> [  566.583974] CPU    9: hi:    0, btch:   1 usd:   0
> [  566.589403] CPU   10: hi:    0, btch:   1 usd:   0
> [  566.594833] CPU   11: hi:    0, btch:   1 usd:   0
> [  566.600262] Node 0 DMA32 per-cpu:
> [  566.604043] CPU    0: hi:  186, btch:  31 usd:  30
> [  566.609473] CPU    1: hi:  186, btch:  31 usd:   0
> [  566.614902] CPU    2: hi:  186, btch:  31 usd:   0
> [  566.620332] CPU    3: hi:  186, btch:  31 usd:  37
> [  566.625760] CPU    4: hi:  186, btch:  31 usd:   0
> [  566.631191] CPU    5: hi:  186, btch:  31 usd:   0
> [  566.636620] CPU    6: hi:  186, btch:  31 usd:   0
> [  566.642050] CPU    7: hi:  186, btch:  31 usd:  30
> [  566.647479] CPU    8: hi:  186, btch:  31 usd:   0
> [  566.652908] CPU    9: hi:  186, btch:  31 usd:   0
> [  566.658337] CPU   10: hi:  186, btch:  31 usd:   0
> [  566.663767] CPU   11: hi:  186, btch:  31 usd:   0
> [  566.669197] Node 0 Normal per-cpu:
> [  566.673075] CPU    0: hi:  186, btch:  31 usd:  30
> [  566.678504] CPU    1: hi:  186, btch:  31 usd:   0
> [  566.683934] CPU    2: hi:  186, btch:  31 usd:   0
> [  566.689362] CPU    3: hi:  186, btch:  31 usd:   0
> [  566.694793] CPU    4: hi:  186, btch:  31 usd:   0
> [  566.700222] CPU    5: hi:  186, btch:  31 usd:   0
> [  566.705651] CPU    6: hi:  186, btch:  31 usd:   0
> [  566.711080] CPU    7: hi:  186, btch:  31 usd:   0
> [  566.716509] CPU    8: hi:  186, btch:  31 usd:   0
> [  566.721939] CPU    9: hi:  186, btch:  31 usd:   0
> [  566.727369] CPU   10: hi:  186, btch:  31 usd:   0
> [  566.732799] CPU   11: hi:  186, btch:  31 usd:   0
> [  566.738230] active_anon:8857 inactive_anon:2282 isolated_anon:448
> [  566.738231]  active_file:1368 inactive_file:1661 isolated_file:0
> [  566.738232]  unevictable:6 dirty:0 writeback:3724 unstable:0
> [  566.738232]  free:30171 slab_reclaimable:1883 slab_unreclaimable:12433
> [  566.738233]  mapped:483 shmem:2 pagetables:921 bounce:0
> [  566.771581] Node 0 DMA free:12124kB min:8kB low:8kB high:12kB 
> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:11968kB 
> mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
> slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB 
> bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> [  566.812205] lowmem_reserve[]: 0 2171 25559 25559
> [  566.817439] Node 0 DMA32 free:94800kB min:1736kB low:2168kB high:2604kB 
> active_anon:35428kB inactive_anon:9128kB active_file:5472kB 
> inactive_file:6644kB unevictable:24kB isolated(anon):1792kB 
> isolated(file):0kB present:2223968kB mlocked:24kB dirty:0kB writeback:14896kB 
> mapped:1932kB shmem:8kB slab_reclaimable:7516kB slab_unreclaimable:48524kB 
> kernel_stack:2648kB pagetables:3684kB unstable:0kB bounce:0kB 
> writeback_tmp:0kB pages_scanned:86560 all_unreclaimable? yes
> [  566.863299] lowmem_reserve[]: 0 0 23387 23387
> [  566.868243] Node 0 Normal free:13760kB min:18720kB low:23400kB 
> high:28080kB active_anon:0kB inactive_anon:0kB active_file:0kB 
> inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB 
> present:23949120kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB 
> slab_reclaimable:16kB slab_unreclaimable:1208kB kernel_stack:0kB 
> pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 
> all_unreclaimable? yes
> [  566.910903] lowmem_reserve[]: 0 0 0 0
> [  566.915115] Node 0 DMA: 1*4kB 1*8kB 1*16kB 2*32kB 2*64kB 1*128kB 2*256kB 
> 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12124kB
> [  566.927040] Node 0 DMA32: 660*4kB 222*8kB 171*16kB 205*32kB 157*64kB 
> 103*128kB 44*256kB 15*512kB 16*1024kB 3*2048kB 4*4096kB = 94800kB
> [  566.940614] Node 0 Normal: 0*4kB 0*8kB 2*16kB 1*32kB 2*64kB 2*128kB 
> 2*256kB 3*512kB 1*1024kB 3*2048kB 1*4096kB = 13760kB
> [  566.952829] 4750 total pagecache pages
> [  566.957096] 1719 pages in swap cache
> [  566.961169] Swap cache stats: add 9971, delete 8252, find 497/815
> [  566.968051] Free swap  = 494992kB
> [  566.971833] Total swap = 524280kB
> [  567.019196] 6660096 pages RAM
> [  567.022406] 6207063 pages reserved
> [  567.026284] 7958 pages shared
> [  567.029678] 418013 pages non-shared
> [  567.033654] Out of memory: kill process 2999 (xend) score 14111 or a child
> [  567.041410] Killed process 3000 (xend)
> 
> 
> 
> (XEN) Physical memory information:
> (XEN)     Xen heap: 0kB free
> (XEN)     heap[14]: 64480kB free
> (XEN)     heap[15]: 131072kB free
> (XEN)     heap[16]: 262144kB free
> (XEN)     heap[17]: 524288kB free
> (XEN)     heap[18]: 1048576kB free
> (XEN)     heap[19]: 196292kB free
> (XEN)     heap[20]: 4194304kB free
> (XEN)     heap[21]: 8388608kB free
> (XEN)     heap[22]: 4479588kB free
> (XEN)     Dom heap: 19289352kB free
> 

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel