[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] unexpected Out Of Memory (OOM)



Le mercredi 07 aoÃt 2013 Ã 16:26 +0200, Olivier Bonvalet a Ãcrit :
> 
> Le mercredi 07 aoÃt 2013 Ã 14:36 +0100, Ian Campbell a Ãcrit :
> > On Wed, 2013-08-07 at 13:17 +0200, Olivier Bonvalet wrote:
> > 
> > > name              = 'reto'
> > > vcpus             = 1
> > > maxvcpus  = 8
> > > memory            = 8192
> > > vif               = [ 'mac=0e:00:00:00:8e:70,bridge=vlan' ]
> > > disk              = [ 
> > >                   '/dev/rbd/sas3copies/reto-root,,xvda,w',
> > >                   '/dev/rbd/sas3copies/reto-home,,xvdb,w',
> > >                   '/dev/rbd/sas3copies/reto-var,,xvdc,w',
> > >                   '/dev/rbd/sas3copies/reto-mysql,,xvdd,w',
> > >                   '/dev/rbd/sas3copies/reto-exim,,xvde,w',
> > > 
> > >             '/dev/loop1,raw,xvdy,r',
> > >             '/dev/loop2,raw,xvdz,r' ]
> > > kernel            = '/etc/xen/kernels/reto/vmlinuz'
> > > ramdisk           = '/etc/xen/kernels/reto/initrd.img'
> > > root              = '/dev/xvda ro rootfstype=ext4'
> > > extra             = 'panic=60'
> > 
> > All looks pretty normal.
> > 
> > > Then the console :
> > > 
> > > Parsing config from /etc/xen/reto.cfg
> > > Daemon running with PID 20283
> > > [    0.000000] Initializing cgroup subsys cpu
> > > [    0.000000] Linux version 2.6.50-dae-xen (root@yiu) (gcc version 4.8.1 
> > > (Debian 4.8.1-8) ) #2 SMP Sun Aug 4 22:42:05 CEST 2013
> > > [    0.000000] Command line: root=/dev/xvda ro rootfstype=ext4 panic=60
> > > [    0.000000] KERNEL supported cpus:
> > > [    0.000000]   Intel GenuineIntel
> > > [    0.000000] ACPI in unprivileged domain disabled
> > > [    0.000000] e820: BIOS-provided physical RAM map:
> > > [    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
> > > [    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
> > > [    0.000000] Xen: [mem 0x0000000000100000-0x00000001ffffffff] usable
> > 
> > This all looks good.
> > 
> > > [    1.776382] blkfront: xvdb: barrier or flush: disabled using 
> > > persistent grants
> > > [    1.797557]  xvdb: unknown partition table
> > > [    1.798526] blkfront: xvdc: barrier or flush: disabled using 
> > > persistent grants
> > > [    1.820815]  xvdc: unknown partition table
> > > [    1.822259] blkfront: xvdd: barrier or flush: disabled using 
> > > persistent grants
> > > [    1.853766]  xvdd: unknown partition table
> > > [    1.854749] blkfront: xvde: barrier or flush: disabled using 
> > > persistent grants
> > > [    1.857572]  xvde: unknown partition table
> > > [    1.858010] Setting capacity to 2097152
> > > [    1.858018] xvda: detected capacity change from 0 to 1073741824
> > > [    1.858760] blkfront: xvdy: flush diskcache: enabled using persistent 
> > > grants
> > > [    1.860885]  xvdy: unknown partition table
> > > [    1.861913] blkfront: xvdz: flush diskcache: enabled using persistent 
> > > grants
> > > [    1.863770]  xvdz: unknown partition table
> > 
> > I know persistent grants have a fixed memory overhead, not sure if that
> > is on the back or frontend though. Roger?
> > 
> > is there any way to disable persistent grants manually for testing
> > purposes?
> > 
> > > [    8.595343] zram: module is from the staging directory, the quality is 
> > > unknown, you have been warned.
> > > [    8.596003] zram: Created 1 device(s) ...
> > 
> > My first thought was that there was a memory leak in the kernel
> > somewhere, but staging drivers doing "magic" things with memory make me
> > nervous. Can you try without zram to rule it out please?
> > 
> > Ian.
> > 
> > 
> 
> You're right, I disabled zram on this Domu (others didn't have zram).
> The problem is still present.
> 
> 
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxx
> http://lists.xen.org/xen-users


So I recompiled a kernel with the kmemleak feature. I obtain that kind
of list, but not sure that it's usefull :

unreferenced object 0xffff88007cca8a80 (size 8):
  comm "swapper/0", pid 1, jiffies 4294892300 (age 150.688s)
  hex dump (first 8 bytes):
    74 69 6d 65 72 31 00 a5                          timer1..
  backtrace:
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
    [<ffffffff811b7071>] kvasprintf+0x41/0x68
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff810056eb>] xen_setup_timer+0x62/0xc3
    [<ffffffff8134296e>] xen_cpu_up+0x5b/0x367
    [<ffffffff81348662>] _cpu_up+0xb8/0x124
    [<ffffffff81348792>] cpu_up+0xc4/0xd0
    [<ffffffff8169dca6>] smp_init+0x4a/0x75
    [<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffff8133d92f>] kernel_init+0x5/0xd1
    [<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007c527d90 (size 16):
  comm "swapper/0", pid 1, jiffies 4294892302 (age 150.680s)
  hex dump (first 16 bytes):
    72 65 73 63 68 65 64 31 00 6b 6b 6b 6b 6b 6b a5  resched1.kkkkkk.
  backtrace:
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
    [<ffffffff811b7071>] kvasprintf+0x41/0x68
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff8100d03e>] text_poke+0x186/0x195
    [<ffffffff8134d595>] xen_smp_intr_init+0x39/0x271
    [<ffffffff81342c28>] xen_cpu_up+0x315/0x367
    [<ffffffff81348662>] _cpu_up+0xb8/0x124
    [<ffffffff81348792>] cpu_up+0xc4/0xd0
    [<ffffffff8169dca6>] smp_init+0x4a/0x75
    [<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffff8133d92f>] kernel_init+0x5/0xd1
    [<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007c526158 (size 16):
  comm "swapper/0", pid 1, jiffies 4294892303 (age 150.676s)
  hex dump (first 16 bytes):
    63 61 6c 6c 66 75 6e 63 31 00 6b 6b 6b 6b 6b a5  callfunc1.kkkkk.
  backtrace:
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
    [<ffffffff811b7071>] kvasprintf+0x41/0x68
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff8134d5db>] xen_smp_intr_init+0x7f/0x271
    [<ffffffff81342c28>] xen_cpu_up+0x315/0x367
    [<ffffffff81348662>] _cpu_up+0xb8/0x124
    [<ffffffff81348792>] cpu_up+0xc4/0xd0
    [<ffffffff8169dca6>] smp_init+0x4a/0x75
    [<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffff8133d92f>] kernel_init+0x5/0xd1
    [<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007cca8bd0 (size 8):
  comm "swapper/0", pid 1, jiffies 4294892303 (age 150.676s)
  hex dump (first 8 bytes):
    64 65 62 75 67 31 00 a5                          debug1..
  backtrace:
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
    [<ffffffff811b7071>] kvasprintf+0x41/0x68
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff8134d624>] xen_smp_intr_init+0xc8/0x271
    [<ffffffff81342c28>] xen_cpu_up+0x315/0x367
    [<ffffffff81348662>] _cpu_up+0xb8/0x124
    [<ffffffff81348792>] cpu_up+0xc4/0xd0
    [<ffffffff8169dca6>] smp_init+0x4a/0x75
    [<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffff8133d92f>] kernel_init+0x5/0xd1
    [<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007c527c38 (size 16):
  comm "swapper/0", pid 1, jiffies 4294892303 (age 150.680s)
  hex dump (first 16 bytes):
    63 61 6c 6c 66 75 6e 63 73 69 6e 67 6c 65 31 00  callfuncsingle1.
  backtrace:
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
    [<ffffffff811b7071>] kvasprintf+0x41/0x68
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff812126ef>] bind_virq_to_irqhandler+0x3c/0x58
    [<ffffffff8134d66d>] xen_smp_intr_init+0x111/0x271
    [<ffffffff81342c28>] xen_cpu_up+0x315/0x367
    [<ffffffff81348662>] _cpu_up+0xb8/0x124
    [<ffffffff81348792>] cpu_up+0xc4/0xd0
    [<ffffffff8169dca6>] smp_init+0x4a/0x75
    [<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffff8133d92f>] kernel_init+0x5/0xd1
    [<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007c5262b0 (size 16):
  comm "swapper/0", pid 1, jiffies 4294892303 (age 150.680s)
  hex dump (first 16 bytes):
    69 72 71 77 6f 72 6b 31 00 6b 6b 6b 6b 6b 6b a5  irqwork1.kkkkkk.
  backtrace:
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
    [<ffffffff811b7071>] kvasprintf+0x41/0x68
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff8134d6c6>] xen_smp_intr_init+0x16a/0x271
    [<ffffffff81342c28>] xen_cpu_up+0x315/0x367
    [<ffffffff81348662>] _cpu_up+0xb8/0x124
    [<ffffffff81348792>] cpu_up+0xc4/0xd0
    [<ffffffff8169dca6>] smp_init+0x4a/0x75
    [<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffff8133d92f>] kernel_init+0x5/0xd1
    [<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007cca93b0 (size 8):
  comm "swapper/0", pid 1, jiffies 4294892303 (age 150.680s)
  hex dump (first 8 bytes):
    74 69 6d 65 72 32 00 a5                          timer2..
  backtrace:
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
    [<ffffffff811b7071>] kvasprintf+0x41/0x68
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff810056eb>] xen_setup_timer+0x62/0xc3
    [<ffffffff8134296e>] xen_cpu_up+0x5b/0x367
    [<ffffffff81348662>] _cpu_up+0xb8/0x124
    [<ffffffff81348792>] cpu_up+0xc4/0xd0
    [<ffffffff8169dca6>] smp_init+0x4a/0x75
    [<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffff8133d92f>] kernel_init+0x5/0xd1
    [<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007c526560 (size 16):
  comm "swapper/0", pid 1, jiffies 4294892303 (age 150.680s)
  hex dump (first 16 bytes):
    73 70 69 6e 6c 6f 63 6b 32 00 6b 6b 6b 6b 6b a5  spinlock2.kkkkk.
  backtrace:
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
    [<ffffffff811b7071>] kvasprintf+0x41/0x68
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff81342cde>] xen_init_lock_cpu+0x64/0xbc
    [<ffffffff81342975>] xen_cpu_up+0x62/0x367
    [<ffffffff81348662>] _cpu_up+0xb8/0x124
    [<ffffffff81348792>] cpu_up+0xc4/0xd0
    [<ffffffff8169dca6>] smp_init+0x4a/0x75
    [<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffff8133d92f>] kernel_init+0x5/0xd1
    [<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007c527830 (size 16):
  comm "swapper/0", pid 1, jiffies 4294892303 (age 150.680s)
  hex dump (first 16 bytes):
    72 65 73 63 68 65 64 32 00 6b 6b 6b 6b 6b 6b a5  resched2.kkkkkk.
  backtrace:
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
    [<ffffffff811b7071>] kvasprintf+0x41/0x68
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff8134d595>] xen_smp_intr_init+0x39/0x271
    [<ffffffff81342c28>] xen_cpu_up+0x315/0x367
    [<ffffffff81348662>] _cpu_up+0xb8/0x124
    [<ffffffff81348792>] cpu_up+0xc4/0xd0
    [<ffffffff8169dca6>] smp_init+0x4a/0x75
    [<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffff8133d92f>] kernel_init+0x5/0xd1
    [<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007c5266b8 (size 16):
  comm "swapper/0", pid 1, jiffies 4294892303 (age 150.680s)
  hex dump (first 16 bytes):
    63 61 6c 6c 66 75 6e 63 32 00 6b 6b 6b 6b 6b a5  callfunc2.kkkkk.
  backtrace:
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
    [<ffffffff811b7071>] kvasprintf+0x41/0x68
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff8134d5db>] xen_smp_intr_init+0x7f/0x271
    [<ffffffff81342c28>] xen_cpu_up+0x315/0x367
    [<ffffffff81348662>] _cpu_up+0xb8/0x124
    [<ffffffff81348792>] cpu_up+0xc4/0xd0
    [<ffffffff8169dca6>] smp_init+0x4a/0x75
    [<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffff8133d92f>] kernel_init+0x5/0xd1
    [<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007cca8d20 (size 8):
  comm "swapper/0", pid 1, jiffies 4294892303 (age 150.680s)
  hex dump (first 8 bytes):
    64 65 62 75 67 32 00 a5                          debug2..
  backtrace:
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
    [<ffffffff811b7071>] kvasprintf+0x41/0x68
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff8134d624>] xen_smp_intr_init+0xc8/0x271
    [<ffffffff81342c28>] xen_cpu_up+0x315/0x367
    [<ffffffff81348662>] _cpu_up+0xb8/0x124
    [<ffffffff81348792>] cpu_up+0xc4/0xd0
    [<ffffffff8169dca6>] smp_init+0x4a/0x75
    [<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffff8133d92f>] kernel_init+0x5/0xd1
    [<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007c5276d8 (size 16):
  comm "swapper/0", pid 1, jiffies 4294892303 (age 150.680s)
  hex dump (first 16 bytes):
    63 61 6c 6c 66 75 6e 63 73 69 6e 67 6c 65 32 00  callfuncsingle2.
  backtrace:
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
    [<ffffffff811b7071>] kvasprintf+0x41/0x68
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff812126ef>] bind_virq_to_irqhandler+0x3c/0x58
    [<ffffffff8134d66d>] xen_smp_intr_init+0x111/0x271
    [<ffffffff81342c28>] xen_cpu_up+0x315/0x367
    [<ffffffff81348662>] _cpu_up+0xb8/0x124
    [<ffffffff81348792>] cpu_up+0xc4/0xd0
    [<ffffffff8169dca6>] smp_init+0x4a/0x75
    [<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffff8133d92f>] kernel_init+0x5/0xd1
    [<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88007c526810 (size 16):
  comm "swapper/0", pid 1, jiffies 4294892303 (age 150.680s)
  hex dump (first 16 bytes):
    69 72 71 77 6f 72 6b 32 00 6b 6b 6b 6b 6b 6b a5  irqwork2.kkkkkk.
  backtrace:
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff810e296d>] __kmalloc_track_caller+0xe7/0xf3
    [<ffffffff811b7071>] kvasprintf+0x41/0x68
    [<ffffffff811b70d6>] kasprintf+0x3e/0x40
    [<ffffffff8134d6c6>] xen_smp_intr_init+0x16a/0x271
    [<ffffffff81342c28>] xen_cpu_up+0x315/0x367
    [<ffffffff81348662>] _cpu_up+0xb8/0x124
    [<ffffffff81348792>] cpu_up+0xc4/0xd0
    [<ffffffff8169dca6>] smp_init+0x4a/0x75
    [<ffffffff81689d55>] kernel_init_freeable+0x78/0x18d
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffff8133d92f>] kernel_init+0x5/0xd1
    [<ffffffff81354efc>] ret_from_fork+0x7c/0xb0
    [<ffffffff8133d92a>] kernel_init+0x0/0xd1
    [<ffffffffffffffff>] 0xffffffffffffffff





_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.