[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] unexpected Out Of Memory (OOM)



I have a DomU which is currently running with 8GB of memory (confirmed
in /proc/meminfo), and only 6GB of memory is used, without process using
it :

root! to:~# free -m
             total       used       free     shared    buffers     cached
Mem:          8024       6642       1381          0         32        252
-/+ buffers/cache:       6357       1667
Swap:          255          0        255



root! to:~# ps auxw
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   8408   800 ?        Ss   Aug06   0:01 init [2]       
root         2  0.0  0.0      0     0 ?        S    Aug06   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    Aug06   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S    Aug06   0:00 [kworker/0:0]
root         5  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kworker/0:0H]
root         7  0.0  0.0      0     0 ?        S    Aug06   0:00 [migration/0]
root         8  0.0  0.0      0     0 ?        S    Aug06   0:00 [rcu_bh]
root         9  0.0  0.0      0     0 ?        S    Aug06   0:00 [rcu_sched]
root        10  0.0  0.0      0     0 ?        S    Aug06   0:00 [watchdog/0]
root        11  0.0  0.0      0     0 ?        R    Aug06   0:00 [watchdog/1]
root        12  0.0  0.0      0     0 ?        R    Aug06   0:00 [migration/1]
root        13  0.0  0.0      0     0 ?        R    Aug06   0:00 [ksoftirqd/1]
root        14  0.0  0.0      0     0 ?        S    Aug06   0:00 [kworker/1:0]
root        15  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kworker/1:0H]
root        16  0.0  0.0      0     0 ?        R    Aug06   0:00 [watchdog/2]
root        17  0.0  0.0      0     0 ?        R    Aug06   0:00 [migration/2]
root        18  0.0  0.0      0     0 ?        R    Aug06   0:00 [ksoftirqd/2]
root        19  0.0  0.0      0     0 ?        S    Aug06   0:00 [kworker/2:0]
root        20  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kworker/2:0H]
root        21  0.0  0.0      0     0 ?        R    Aug06   0:00 [watchdog/3]
root        22  0.0  0.0      0     0 ?        R    Aug06   0:00 [migration/3]
root        23  0.0  0.0      0     0 ?        R    Aug06   0:00 [ksoftirqd/3]
root        24  0.0  0.0      0     0 ?        S    Aug06   0:00 [kworker/3:0]
root        25  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kworker/3:0H]
root        26  0.0  0.0      0     0 ?        R    Aug06   0:00 [watchdog/4]
root        27  0.0  0.0      0     0 ?        R    Aug06   0:00 [migration/4]
root        28  0.0  0.0      0     0 ?        R    Aug06   0:00 [ksoftirqd/4]
root        29  0.0  0.0      0     0 ?        S    Aug06   0:00 [kworker/4:0]
root        30  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kworker/4:0H]
root        31  0.0  0.0      0     0 ?        R    Aug06   0:00 [watchdog/5]
root        32  0.0  0.0      0     0 ?        R    Aug06   0:00 [migration/5]
root        33  0.0  0.0      0     0 ?        R    Aug06   0:00 [ksoftirqd/5]
root        34  0.0  0.0      0     0 ?        S    Aug06   0:00 [kworker/5:0]
root        35  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kworker/5:0H]
root        36  0.0  0.0      0     0 ?        R    Aug06   0:00 [watchdog/6]
root        37  0.0  0.0      0     0 ?        R    Aug06   0:00 [migration/6]
root        38  0.0  0.0      0     0 ?        R    Aug06   0:00 [ksoftirqd/6]
root        39  0.0  0.0      0     0 ?        S    Aug06   0:00 [kworker/6:0]
root        40  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kworker/6:0H]
root        41  0.0  0.0      0     0 ?        R    Aug06   0:00 [watchdog/7]
root        42  0.0  0.0      0     0 ?        R    Aug06   0:00 [migration/7]
root        43  0.0  0.0      0     0 ?        R    Aug06   0:00 [ksoftirqd/7]
root        44  0.0  0.0      0     0 ?        S    Aug06   0:00 [kworker/7:0]
root        45  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kworker/7:0H]
root        46  0.0  0.0      0     0 ?        S<   Aug06   0:00 [khelper]
root        47  0.0  0.0      0     0 ?        S    Aug06   0:00 [kdevtmpfs]
root        48  0.0  0.0      0     0 ?        S    Aug06   0:00 [xenwatch]
root        49  0.0  0.0      0     0 ?        S    Aug06   0:00 [xenbus]
root        50  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kworker/1:1H]
root        51  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kworker/2:1H]
root        52  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kworker/3:1H]
root        53  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kworker/4:1H]
root        54  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kworker/5:1H]
root        55  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kworker/6:1H]
root        56  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kworker/7:1H]
root        57  0.0  0.0      0     0 ?        S<   Aug06   0:00 [writeback]
root        58  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kintegrityd]
root        59  0.0  0.0      0     0 ?        S<   Aug06   0:00 [bioset]
root        60  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kblockd]
root        61  0.0  0.0      0     0 ?        S<   Aug06   0:00 
[xenbus_frontend]
root        62  0.0  0.0      0     0 ?        S<   Aug06   0:00 [ata_sff]
root        63  0.0  0.0      0     0 ?        S<   Aug06   0:00 [edac-poller]
root        64  0.0  0.0      0     0 ?        S    Aug06   0:02 [kworker/0:1]
root       106  0.0  0.0      0     0 ?        S    Aug06   0:00 [kswapd0]
root       107  0.0  0.0      0     0 ?        SN   Aug06   0:00 [ksmd]
root       108  0.0  0.0      0     0 ?        S    Aug06   0:00 [fsnotify_mark]
root       109  0.0  0.0      0     0 ?        S<   Aug06   0:00 [crypto]
root       126  0.0  0.0      0     0 ?        S    Aug06   0:00 [khvcd]
root       127  0.0  0.0      0     0 ?        S<   Aug06   0:00 [iscsi_eh]
root       128  0.0  0.0      0     0 ?        S<   Aug06   0:00 [deferwq]
root       129  0.0  0.0      0     0 ?        S    Aug06   0:00 [kworker/u16:1]
root       253  0.0  0.0      0     0 ?        S    Aug06   0:00 [jbd2/xvda-8]
root       254  0.0  0.0      0     0 ?        S<   Aug06   0:00 
[ext4-dio-unwrit]
root       257  0.0  0.0      0     0 ?        S<   Aug06   0:00 
[ext4-dio-unwrit]
root       259  0.0  0.0      0     0 ?        S<   Aug06   0:00 
[ext4-dio-unwrit]
root       520  0.0  0.0      0     0 ?        S<   Aug06   0:00 [kdmflush]
root       704  0.0  0.0      0     0 ?        S    Aug06   0:00 [jbd2/xvdc-8]
root       706  0.0  0.0      0     0 ?        S<   Aug06   0:00 
[ext4-dio-unwrit]
root       707  0.0  0.0      0     0 ?        S    Aug06   0:00 [jbd2/xvdb-8]
root       708  0.0  0.0      0     0 ?        S<   Aug06   0:00 
[ext4-dio-unwrit]
root       709  0.0  0.0      0     0 ?        S    Aug06   0:00 [jbd2/dm-0-8]
root       710  0.0  0.0      0     0 ?        S<   Aug06   0:00 
[ext4-dio-unwrit]
root       713  0.0  0.0      0     0 ?        S<   Aug06   0:00 [bioset]
root       714  0.0  0.0      0     0 ?        S    Aug06   0:00 
[btrfs-worker-1]
root       715  0.0  0.0      0     0 ?        S    Aug06   0:00 
[btrfs-genwork-1]
root       716  0.0  0.0      0     0 ?        S    Aug06   0:00 
[btrfs-submit-1]
root       717  0.0  0.0      0     0 ?        S    Aug06   0:00 
[btrfs-delalloc-]
root       718  0.0  0.0      0     0 ?        S    Aug06   0:00 [btrfs-fixup-1]
root       719  0.0  0.0      0     0 ?        S    Aug06   0:00 [btrfs-endio-1]
root       720  0.0  0.0      0     0 ?        S    Aug06   0:00 
[btrfs-endio-met]
root       721  0.0  0.0      0     0 ?        S    Aug06   0:00 [btrfs-rmw-1]
root       722  0.0  0.0      0     0 ?        S    Aug06   0:00 
[btrfs-endio-rai]
root       723  0.0  0.0      0     0 ?        S    Aug06   0:00 
[btrfs-endio-met]
root       724  0.0  0.0      0     0 ?        S    Aug06   0:00 
[btrfs-endio-wri]
root       725  0.0  0.0      0     0 ?        S    Aug06   0:00 
[btrfs-freespace]
root       726  0.0  0.0      0     0 ?        S    Aug06   0:00 
[btrfs-delayed-m]
root       727  0.0  0.0      0     0 ?        S    Aug06   0:00 [btrfs-cache-1]
root       728  0.0  0.0      0     0 ?        S    Aug06   0:00 
[btrfs-readahead]
root       729  0.0  0.0      0     0 ?        S    Aug06   0:00 
[btrfs-flush_del]
root       730  0.0  0.0      0     0 ?        S    Aug06   0:00 
[btrfs-qgroup-re]
root       731  0.0  0.0      0     0 ?        S    Aug06   0:00 [btrfs-cleaner]
root       732  0.0  0.0      0     0 ?        S    Aug06   0:00 
[btrfs-transacti]
root      1845  0.0  0.0   5984   620 tty1     Ss+  Aug06   0:00 /sbin/getty 
38400 tty1
root      1846  0.0  0.0   5984   624 hvc0     Ss+  Aug06   0:00 /sbin/getty 
38400 hvc0
root      1908  0.0  0.0      0     0 ?        S    Aug06   0:00 [kworker/u16:2]
root      6457  0.0  0.0  76740  3348 ?        Ss   11:23   0:00 sshd: daevel 
[priv]
daevel    6460  0.0  0.0  76740  1928 ?        S    11:23   0:00 sshd: 
daevel@pts/0,pts/1
daevel    6461  0.0  0.0  25040  7312 pts/0    Ss   11:23   0:00 -bash
daevel    6640  0.0  0.0  25036  7296 pts/1    Ss   11:23   0:00 -bash
root      6668  0.0  0.0  25128  1264 pts/1    S    11:23   0:00 sudo su
root      6669  0.0  0.0  45804  1348 pts/1    S    11:23   0:00 su
root      6671  0.0  0.0  19944  2168 pts/1    S    11:23   0:00 bash
daevel    8314  0.0  0.0   5924   560 pts/0    S+   11:32   0:00 sleep 30
root      8315  0.0  0.0  16900  1176 pts/1    R+   11:33   0:00 ps auxw
root     11919  0.0  0.0      0     0 ?        S<   02:00   0:00 [kworker/0:1H]
root     11971  0.0  0.0      0     0 ?        S<   02:00   0:00 [bioset]
root! to:~# 

root! to:~# cat /proc/meminfo 
MemTotal:        8217004 kB
MemFree:         1415044 kB
Buffers:           33128 kB
Cached:           258884 kB
SwapCached:            0 kB
Active:           123328 kB
Inactive:         182692 kB
Active(anon):      14032 kB
Inactive(anon):      344 kB
Active(file):     109296 kB
Inactive(file):   182348 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:        262140 kB
SwapFree:         262140 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:         14024 kB
Mapped:             4300 kB
Shmem:               396 kB
Slab:            5105244 kB
SReclaimable:    1563800 kB
SUnreclaim:      3541444 kB
KernelStack:     1360232 kB
PageTables:          860 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     4370640 kB
Committed_AS:      20612 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       16248 kB
VmallocChunk:   34359721447 kB
AnonHugePages:         0 kB
DirectMap4k:     8388608 kB
DirectMap2M:           0 kB
root! to:~# 


root! to:~# slabtop --once ; sleep 30 ; echo ; slabtop --once
 Active / Total Objects (% used)    : 17298069 / 17301644 (100.0%)
 Active / Total Slabs (% used)      : 597110 / 597110 (100.0%)
 Active / Total Caches (% used)     : 72 / 93 (77.4%)
 Active / Total Size (% used)       : 5045185.95K / 5045752.48K (100.0%)
 Minimum / Average / Maximum Object : 0.01K / 0.29K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
10290816 10290816 100%    0.25K 321588       32   2572704K kmalloc-256          
  
1997499 1997499 100%    0.19K  95119       21    380476K dentry                 
1904890 1904890 100%    0.59K  73265       26   1172240K proc_inode_cache       
1305654 1305654 100%    0.19K  62174       21    248696K kmalloc-192            
863488 861305  99%    0.06K  13492       64     53968K kmalloc-64             
180540 180540 100%    0.11K   5015       36     20060K sysfs_dir_cache        
171808 170841  99%    0.12K   5369       32     21476K kmalloc-128            
170079 170079 100%    0.81K   4361       39    139552K task_xstate            
170050 170050 100%    1.66K   8950       19    286400K task_struct            
169860 169860 100%    1.06K   5662       30    181184K signal_cache           
 19925  19925 100%    0.62K    797       25     12752K shmem_inode_cache      
 19383  19383 100%    0.10K    497       39      1988K buffer_head            
  4608   4608 100%    0.01K      9      512        36K kmalloc-8              
  4592   4592 100%    0.55K    164       28      2624K radix_tree_node        
  4480   4480 100%    0.91K    128       35      4096K ext4_inode_cache       
  3660   3660 100%    0.53K    122       30      1952K inode_cache            
  3200   3200 100%    0.03K     25      128       100K kmalloc-32             
  2816   2816 100%    0.02K     11      256        44K kmalloc-16             
  2448   2448 100%    0.04K     24      102        96K ext4_extent_status     
  2112   2112 100%    0.06K     33       64       132K anon_vma               
  1632   1632 100%    0.04K     16      102        64K ext4_system_zone       
  1298   1186  91%    0.18K     59       22       236K vm_area_struct         
  1024    910  88%    0.50K     32       32       512K kmalloc-512            
   546    546 100%    0.09K     13       42        52K kmalloc-96             
   345    345 100%    0.34K     15       23       120K blkdev_requests        
   340    340 100%    0.05K      4       85        16K shared_policy_node     
   336    336 100%    0.19K     16       21        64K tw_sock_TCP            
   324    324 100%    0.11K      9       36        36K jbd2_journal_head      
   320    320 100%    1.00K     10       32       320K kmalloc-1024           
   312    312 100%    0.10K      8       39        32K flow_cache             
   256    256 100%    0.02K      1      256         4K jbd2_revoke_table_s    
   232     89  38%    0.27K      8       29        64K 
nf_conntrack_ffffffff8165cc40
   195    195 100%    2.06K     13       15       416K idr_layer_cache        
   195    179  91%    2.06K     13       15       416K sighand_cache          
   192    192 100%    0.06K      3       64        12K ext4_io_end            
   170    170 100%    0.02K      1      170         4K numa_policy            
   128    128 100%    0.06K      2       64         8K kmem_cache_node        
   128    128 100%    0.03K      1      128         4K jbd2_revoke_record_s   
   125    125 100%    0.62K      5       25        80K files_cache            
   125    125 100%    0.16K      5       25        20K sigqueue               
   120     80  66%    4.00K     15        8       480K kmalloc-4096           
   112    112 100%    2.00K      7       16       224K kmalloc-2048           
   100    100 100%    0.62K      4       25        64K sock_inode_cache       
    96     96 100%    0.25K      3       32        24K kmem_cache             
    96     96 100%    0.98K      3       32        96K btrfs_inode            

 Active / Total Objects (% used)    : 17298196 / 17301771 (100.0%)
 Active / Total Slabs (% used)      : 597115 / 597115 (100.0%)
 Active / Total Caches (% used)     : 72 / 93 (77.4%)
 Active / Total Size (% used)       : 5045213.76K / 5045780.30K (100.0%)
 Minimum / Average / Maximum Object : 0.01K / 0.29K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
10290880 10290880 100%    0.25K 321590       32   2572720K kmalloc-256          
  
1997520 1997520 100%    0.19K  95120       21    380480K dentry                 
1904890 1904890 100%    0.59K  73265       26   1172240K proc_inode_cache       
1305696 1305696 100%    0.19K  62176       21    248704K kmalloc-192            
863488 861305  99%    0.06K  13492       64     53968K kmalloc-64             
180540 180540 100%    0.11K   5015       36     20060K sysfs_dir_cache        
171808 170841  99%    0.12K   5369       32     21476K kmalloc-128            
170079 170079 100%    0.81K   4361       39    139552K task_xstate            
170050 170050 100%    1.66K   8950       19    286400K task_struct            
169860 169860 100%    1.06K   5662       30    181184K signal_cache           
 19925  19925 100%    0.62K    797       25     12752K shmem_inode_cache      
 19383  19383 100%    0.10K    497       39      1988K buffer_head            
  4608   4608 100%    0.01K      9      512        36K kmalloc-8              
  4592   4592 100%    0.55K    164       28      2624K radix_tree_node        
  4480   4480 100%    0.91K    128       35      4096K ext4_inode_cache       
  3660   3660 100%    0.53K    122       30      1952K inode_cache            
  3200   3200 100%    0.03K     25      128       100K kmalloc-32             
  2816   2816 100%    0.02K     11      256        44K kmalloc-16             
  2448   2448 100%    0.04K     24      102        96K ext4_extent_status     
  2112   2112 100%    0.06K     33       64       132K anon_vma               
  1632   1632 100%    0.04K     16      102        64K ext4_system_zone       
  1298   1186  91%    0.18K     59       22       236K vm_area_struct         
  1024    910  88%    0.50K     32       32       512K kmalloc-512            
   546    546 100%    0.09K     13       42        52K kmalloc-96             
   345    345 100%    0.34K     15       23       120K blkdev_requests        
   340    340 100%    0.05K      4       85        16K shared_policy_node     
   336    336 100%    0.19K     16       21        64K tw_sock_TCP            
   324    324 100%    0.11K      9       36        36K jbd2_journal_head      
   320    320 100%    1.00K     10       32       320K kmalloc-1024           
   312    312 100%    0.10K      8       39        32K flow_cache             
   256    256 100%    0.02K      1      256         4K jbd2_revoke_table_s    
   232     89  38%    0.27K      8       29        64K 
nf_conntrack_ffffffff8165cc40
   195    195 100%    2.06K     13       15       416K idr_layer_cache        
   195    179  91%    2.06K     13       15       416K sighand_cache          
   192    192 100%    0.06K      3       64        12K ext4_io_end            
   170    170 100%    0.02K      1      170         4K numa_policy            
   128    128 100%    0.06K      2       64         8K kmem_cache_node        
   128    128 100%    0.03K      1      128         4K jbd2_revoke_record_s   
   125    125 100%    0.62K      5       25        80K files_cache            
   125    125 100%    0.16K      5       25        20K sigqueue               
   120     80  66%    4.00K     15        8       480K kmalloc-4096           
   112    112 100%    2.00K      7       16       224K kmalloc-2048           
   100    100 100%    0.62K      4       25        64K sock_inode_cache       
    96     96 100%    0.25K      3       32        24K kmem_cache             
    96     96 100%    0.98K      3       32        96K btrfs_inode            
root! to:~# 








_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.