|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] [GIT PULL] (xen) stable/for-jens-3.10 xenwatch: page allocation failure: order:7, mode:0x10c0d0
Friday, April 19, 2013, 4:44:01 PM, you wrote:
> Hey Jens,
> Please in your spare time (if there is such a thing at a conference)
> pull this branch:
> git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
> stable/for-jens-3.10
> for your v3.10 branch. Sorry for being so late with this.
<big snip></big snip>
> Anyhow, please pull and if possible include the nice overview I typed up in
> the
> merge commit.
> Documentation/ABI/stable/sysfs-bus-xen-backend | 18 +
> drivers/block/xen-blkback/blkback.c | 843
> ++++++++++++++++---------
> drivers/block/xen-blkback/common.h | 145 ++++-
> drivers/block/xen-blkback/xenbus.c | 38 ++
> drivers/block/xen-blkfront.c | 490 +++++++++++---
> include/xen/interface/io/blkif.h | 53 ++
> 6 files changed, 1188 insertions(+), 399 deletions(-)
> Roger Pau Monne (7):
> xen-blkback: print stats about persistent grants
> xen-blkback: use balloon pages for all mappings
> xen-blkback: implement LRU mechanism for persistent grants
> xen-blkback: move pending handles list from blkbk to pending_req
> xen-blkback: make the queue of free requests per backend
> xen-blkback: expand map/unmap functions
> xen-block: implement indirect descriptors
Hi Konrad / Roger,
I tried this pull on top of latest Linus latest linux-3.9 tree, but although it
seems to boot and work fine at first, i seem to get trouble after running for
about a day.
Without this pull it runs fine for several days.
Trying to start a new guest I ended up with the splat below. In the output of
xl-dmesg i seem to see more of these than before:
(XEN) [2013-04-24 14:37:40] grant_table.c:1250:d1 Expanding dom (1) grant table
from (9) to (10) frames
Attached xl-dmesg and dmesg.
--
Sander
[18496.013743] xenwatch: page allocation failure: order:7, mode:0x10c0d0
[18496.031948] Pid: 54, comm: xenwatch Not tainted 3.9.0-rc8-20130424-jens+ #1
[18496.049897] Call Trace:
[18496.067674] [<ffffffff81100c51>] warn_alloc_failed+0xf1/0x140
[18496.085453] [<ffffffff810b25ed>] ? trace_hardirqs_on+0xd/0x10
[18496.102951] [<ffffffff810bdb24>] ? on_each_cpu_mask+0x94/0xd0
[18496.120270] [<ffffffff811028af>] __alloc_pages_nodemask+0x69f/0x960
[18496.137306] [<ffffffff8113a161>] alloc_pages_current+0xb1/0x160
[18496.154051] [<ffffffff81100679>] __get_free_pages+0x9/0x40
[18496.170579] [<ffffffff81142af4>] __kmalloc+0x134/0x160
[18496.186921] [<ffffffff815832d0>] xen_blkbk_probe+0x170/0x2f0
[18496.202963] [<ffffffff81474ce7>] xenbus_dev_probe+0x77/0x130
[18496.218714] [<ffffffff8156a390>] ? __driver_attach+0xa0/0xa0
[18496.234237] [<ffffffff8156a151>] driver_probe_device+0x81/0x220
[18496.249605] [<ffffffff8198198c>] ? klist_next+0x8c/0x110
[18496.264681] [<ffffffff8156a390>] ? __driver_attach+0xa0/0xa0
[18496.279500] [<ffffffff8156a3db>] __device_attach+0x4b/0x50
[18496.294138] [<ffffffff815684e8>] bus_for_each_drv+0x68/0x90
[18496.308553] [<ffffffff8156a0c9>] device_attach+0x89/0x90
[18496.322694] [<ffffffff81569258>] bus_probe_device+0xa8/0xd0
[18496.336640] [<ffffffff81567c80>] device_add+0x650/0x720
[18496.350209] [<ffffffff81573103>] ? device_pm_sleep_init+0x43/0x70
[18496.363602] [<ffffffff81567d69>] device_register+0x19/0x20
[18496.376721] [<ffffffff8147495b>] xenbus_probe_node+0x14b/0x160
[18496.389611] [<ffffffff815685b4>] ? bus_for_each_dev+0xa4/0xb0
[18496.402298] [<ffffffff81474b2c>] xenbus_dev_changed+0x1bc/0x1c0
[18496.414732] [<ffffffff810b67f7>] ? lock_release+0x117/0x260
[18496.426904] [<ffffffff81474f66>] backend_changed+0x16/0x20
[18496.438835] [<ffffffff81472f5e>] xenwatch_thread+0x4e/0x150
[18496.450579] [<ffffffff8108abb0>] ? wake_up_bit+0x40/0x40
[18496.462048] [<ffffffff81472f10>] ? xs_watch+0x60/0x60
[18496.473286] [<ffffffff8108a546>] kthread+0xd6/0xe0
[18496.484235] [<ffffffff8108a470>] ? __init_kthread_worker+0x70/0x70
[18496.494987] [<ffffffff819979bc>] ret_from_fork+0x7c/0xb0
[18496.505560] [<ffffffff8108a470>] ? __init_kthread_worker+0x70/0x70
[18496.515949] Mem-Info:
[18496.526267] Node 0 DMA per-cpu:
[18496.536085] CPU 0: hi: 0, btch: 1 usd: 0
[18496.545692] CPU 1: hi: 0, btch: 1 usd: 0
[18496.554949] CPU 2: hi: 0, btch: 1 usd: 0
[18496.563969] CPU 3: hi: 0, btch: 1 usd: 0
[18496.572690] CPU 4: hi: 0, btch: 1 usd: 0
[18496.581067] CPU 5: hi: 0, btch: 1 usd: 0
[18496.589120] Node 0 DMA32 per-cpu:
[18496.596950] CPU 0: hi: 186, btch: 31 usd: 62
[18496.604583] CPU 1: hi: 186, btch: 31 usd: 85
[18496.611890] CPU 2: hi: 186, btch: 31 usd: 63
[18496.618950] CPU 3: hi: 186, btch: 31 usd: 84
[18496.625743] CPU 4: hi: 186, btch: 31 usd: 100
[18496.632205] CPU 5: hi: 186, btch: 31 usd: 32
[18496.638361] active_anon:2681 inactive_anon:11161 isolated_anon:0
[18496.638361] active_file:30213 inactive_file:148645 isolated_file:0
[18496.638361] unevictable:540 dirty:509 writeback:0 unstable:0
[18496.638361] free:4756 slab_reclaimable:10388 slab_unreclaimable:10266
[18496.638361] mapped:4093 shmem:316 pagetables:1177 bounce:0
[18496.638361] free_cma:0
[18496.670912] Node 0 DMA free:3864kB min:64kB low:80kB high:96kB
active_anon:0kB inactive_anon:0kB active_file:140kB inactive_file:9440kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB
managed:15908kB mlocked:0kB dirty:60kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:2168kB slab_unreclaimable:256kB kernel_stack:8kB
pagetables:4kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? no
[18496.686284] lowmem_reserve[]: 0 884 884 884
[18496.691452] Node 0 DMA32 free:15184kB min:3772kB low:4712kB high:5656kB
active_anon:10612kB inactive_anon:44648kB active_file:120756kB
inactive_file:585244kB unevictable:2068kB isolated(anon):0kB isolated(file):0kB
present:1032192kB managed:905896kB mlocked:2068kB dirty:1988kB writeback:0kB
mapped:16232kB shmem:1268kB slab_reclaimable:39384kB slab_unreclaimable:40812kB
kernel_stack:2176kB pagetables:4684kB unstable:0kB bounce:0kB free_cma:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[18496.714687] lowmem_reserve[]: 0 0 0 0
[18496.720726] Node 0 DMA: 18*4kB (UM) 16*8kB (UM) 15*16kB (UM) 21*32kB (UMR)
19*64kB (UMR) 4*128kB (MR) 4*256kB (R) 0*512kB 0*1024kB 0*2048kB 0*4096kB =
3864kB
[18496.727233] Node 0 DMA32: 2014*4kB (UEMR) 549*8kB (UEMR) 39*16kB (EMR)
14*32kB (ER) 10*64kB (R) 4*128kB (R) 2*256kB (R) 0*512kB 0*1024kB 0*2048kB
0*4096kB = 15184kB
[18496.740356] 179631 total pagecache pages
[18496.746941] 0 pages in swap cache
[18496.753528] Swap cache stats: add 142, delete 142, find 67/73
[18496.760194] Free swap = 2097148kB
[18496.766821] Total swap = 2097148kB
[18496.777073] 262143 pages RAM
[18496.783661] 28027 pages reserved
[18496.790185] 381067 pages shared
[18496.796701] 119219 pages non-shared
[18496.803221] vbd vbd-18-768: 12 creating block interface
[18496.811469] vbd vbd-18-768: 12 xenbus_dev_probe on backend/vbd/18/768
[18496.818901] vbd: probe of vbd-18-768 failed with error -12
#:/mnt/backup# vmstat -a
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free inact active si so bi bo in cs us sy id wa
0 1 0 14116 642816 131760 0 0 297 254 334 383 1 3 89 7
#:/mnt/backup# vmstat -s
936464 K total memory
924468 K used memory
131572 K active memory
645052 K inactive memory
11996 K free memory
123128 K buffer memory
601596 K swap cache
2097148 K total swap
0 K used swap
2097148 K free swap
70635 non-nice user cpu ticks
22910 nice user cpu ticks
277581 system cpu ticks
9765363 idle cpu ticks
769650 IO-wait cpu ticks
208 IRQ cpu ticks
43718 softirq cpu ticks
51929 stolen cpu ticks
32661673 pages paged in
27922685 pages paged out
41 pages swapped in
93 pages swapped out
36733683 interrupts
42159895 CPU context switches
1366806417 boot time
252828 forks
#:/mnt/backup# vmstat -m
Cache Num Total Size Pages
ext4_groupinfo_4k 23382 23382 216 18
xt_hashlimit 0 0 136 30
nf_conntrack_ffffffff81ec7b80 1486 1620 400 20
nf_conntrack_expect 0 0 288 28
dm_snap_pending_exception 0 0 104 39
kcopyd_job 0 0 4176 7
dm_rq_target_io 0 0 416 19
blkif_cache 100 100 800 20
cfq_io_cq 204 204 120 34
cfq_queue 153 153 232 17
bsg_cmd 0 0 312 26
ceph_cap 0 0 128 32
ceph_inode_info 0 0 2032 16
gfs2_mblk 0 0 432 18
gfs2_bufdata 11048 11224 88 46
gfs2_inode 0 0 1280 25
gfs2_glock(aspace) 0 0 960 17
gfs2_glock 0 0 592 27
btrfs_delayed_data_ref 0 0 96 42
btrfs_delayed_ref_head 0 0 232 17
btrfs_delayed_node 0 0 392 20
btrfs_ordered_extent 0 0 464 17
btrfs_extent_buffer 0 0 584 28
btrfs_delalloc_work 0 0 184 22
btrfs_path 0 0 144 28
btrfs_transaction 0 0 656 24
btrfs_trans_handle 0 0 160 25
btrfs_inode 0 0 1944 16
fuse_request 0 0 472 17
fuse_inode 0 0 1152 28
ntfs_big_inode_cache 0 0 1664 19
ntfs_inode_cache 0 0 696 23
cifs_small_rq 36 36 448 18
cifs_request 4 4 16512 1
cifs_inode_cache 0 0 1104 29
isofs_inode_cache 0 0 960 17
fat_inode_cache 0 0 1168 28
fat_cache 0 0 40 102
hugetlbfs_inode_cache 16 16 976 16
jbd2_transaction_s 375 375 320 25
jbd2_journal_handle 336 336 72 56
journal_handle 0 0 56 73
journal_head 1296 1296 112 36
revoke_record 384 384 32 128
ext4_inode_cache 4598 8137 1696 19
ext4_free_data 1408 1408 64 64
ext4_allocation_context 180 180 136 30
ext4_prealloc_space 156 156 152 26
ext4_io_end 390 420 1088 30
ext4_io_page 4096 4096 16 256
ext4_extent_status 8576 8772 40 102
ext3_inode_cache 0 0 1312 24
ext3_xattr 0 0 88 46
dquot 0 0 384 21
dio 92 92 704 23
pid_namespace 0 0 2200 14
posix_timers_cache 116 116 280 29
UNIX 176 176 1472 22
UDP-Lite 0 0 1280 25
ip_fib_trie 292 292 56 73
Cache Num Total Size Pages
RAW 156 156 1216 26
UDP 150 150 1280 25
tw_sock_TCP 378 378 192 21
TCP 117 117 2368 13
fscache_cookie_jar 0 0 184 22
sgpool-128 68 90 5120 6
sgpool-64 72 72 2560 12
sgpool-32 175 175 1280 25
sgpool-16 275 275 640 25
blkdev_integrity 0 0 112 36
blkdev_queue 143 143 2808 11
blkdev_requests 346 346 376 21
fsnotify_event_holder 0 0 24 170
bip-256 7 7 4224 7
bip-128 0 0 2176 15
bip-16 0 0 384 21
sock_inode_cache 323 323 960 17
file_lock_cache 102 102 240 17
net_namespace 0 0 1984 16
shmem_inode_cache 1736 1736 1144 28
Acpi-State 306 306 80 51
Acpi-Namespace 1632 1632 40 102
task_delay_info 960 960 168 24
taskstats 144 144 328 24
proc_inode_cache 1440 1440 976 16
sigqueue 300 300 160 25
bdev_cache 144 144 1344 24
sysfs_dir_cache 22512 22512 144 28
filp 2134 2450 320 25
inode_cache 1260 1972 912 17
dentry 7006 11888 248 16
buffer_head 104786 107328 104 39
vm_area_struct 4131 4444 184 22
mm_struct 280 280 1152 28
files_cache 273 273 768 21
signal_cache 483 529 1408 23
sighand_cache 387 406 2240 14
task_struct 297 329 4208 7
anon_vma 4147 4536 144 28
shared_policy_node 510 510 48 85
numa_policy 3360 3360 72 56
radix_tree_node 5470 5740 568 28
idr_layer_cache 330 330 2112 15
dma-kmalloc-8192 0 0 8192 4
dma-kmalloc-4096 0 0 4096 8
dma-kmalloc-2048 0 0 2048 16
dma-kmalloc-1024 0 0 1024 16
dma-kmalloc-512 0 0 512 16
dma-kmalloc-256 0 0 256 16
dma-kmalloc-128 0 0 128 32
dma-kmalloc-64 0 0 64 64
dma-kmalloc-32 0 0 32 128
dma-kmalloc-16 0 0 16 256
dma-kmalloc-8 0 0 8 512
dma-kmalloc-192 0 0 192 21
dma-kmalloc-96 0 0 96 42
kmalloc-8192 28 28 8192 4
kmalloc-4096 909 928 4096 8
kmalloc-2048 646 704 2048 16
kmalloc-1024 1803 1808 1024 16
Cache Num Total Size Pages
kmalloc-512 1057 1152 512 16
kmalloc-256 4175 4304 256 16
kmalloc-128 2948 3456 128 32
kmalloc-64 13726 15040 64 64
kmalloc-32 1536 1536 32 128
kmalloc-16 4096 4096 16 256
kmalloc-8 7168 7168 8 512
kmalloc-192 68687 69111 192 21
kmalloc-96 3704 4032 96 42
kmem_cache_node 192 192 128 32
kmem_cache 160 160 256 16
# vmstat -a
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free inact active si so bi bo in cs us sy id wa
0 1 0 14116 642816 131760 0 0 297 254 334 383 1 3 89 7
# vmstat -s
936464 K total memory
924468 K used memory
131572 K active memory
645052 K inactive memory
11996 K free memory
123128 K buffer memory
601596 K swap cache
2097148 K total swap
0 K used swap
2097148 K free swap
70635 non-nice user cpu ticks
22910 nice user cpu ticks
277581 system cpu ticks
9765363 idle cpu ticks
769650 IO-wait cpu ticks
208 IRQ cpu ticks
43718 softirq cpu ticks
51929 stolen cpu ticks
32661673 pages paged in
27922685 pages paged out
41 pages swapped in
93 pages swapped out
36733683 interrupts
42159895 CPU context switches
1366806417 boot time
252828 forks
# vmstat -m
Cache Num Total Size Pages
ext4_groupinfo_4k 23382 23382 216 18
xt_hashlimit 0 0 136 30
nf_conntrack_ffffffff81ec7b80 1486 1620 400 20
nf_conntrack_expect 0 0 288 28
dm_snap_pending_exception 0 0 104 39
kcopyd_job 0 0 4176 7
dm_rq_target_io 0 0 416 19
blkif_cache 100 100 800 20
cfq_io_cq 204 204 120 34
cfq_queue 153 153 232 17
bsg_cmd 0 0 312 26
ceph_cap 0 0 128 32
ceph_inode_info 0 0 2032 16
gfs2_mblk 0 0 432 18
gfs2_bufdata 11048 11224 88 46
gfs2_inode 0 0 1280 25
gfs2_glock(aspace) 0 0 960 17
gfs2_glock 0 0 592 27
btrfs_delayed_data_ref 0 0 96 42
btrfs_delayed_ref_head 0 0 232 17
btrfs_delayed_node 0 0 392 20
btrfs_ordered_extent 0 0 464 17
btrfs_extent_buffer 0 0 584 28
btrfs_delalloc_work 0 0 184 22
btrfs_path 0 0 144 28
btrfs_transaction 0 0 656 24
btrfs_trans_handle 0 0 160 25
btrfs_inode 0 0 1944 16
fuse_request 0 0 472 17
fuse_inode 0 0 1152 28
ntfs_big_inode_cache 0 0 1664 19
ntfs_inode_cache 0 0 696 23
cifs_small_rq 36 36 448 18
cifs_request 4 4 16512 1
cifs_inode_cache 0 0 1104 29
isofs_inode_cache 0 0 960 17
fat_inode_cache 0 0 1168 28
fat_cache 0 0 40 102
hugetlbfs_inode_cache 16 16 976 16
jbd2_transaction_s 375 375 320 25
jbd2_journal_handle 336 336 72 56
journal_handle 0 0 56 73
journal_head 1296 1296 112 36
revoke_record 384 384 32 128
ext4_inode_cache 4598 8137 1696 19
ext4_free_data 1408 1408 64 64
ext4_allocation_context 180 180 136 30
ext4_prealloc_space 156 156 152 26
ext4_io_end 390 420 1088 30
ext4_io_page 4096 4096 16 256
ext4_extent_status 8576 8772 40 102
ext3_inode_cache 0 0 1312 24
ext3_xattr 0 0 88 46
dquot 0 0 384 21
dio 92 92 704 23
pid_namespace 0 0 2200 14
posix_timers_cache 116 116 280 29
UNIX 176 176 1472 22
UDP-Lite 0 0 1280 25
ip_fib_trie 292 292 56 73
Cache Num Total Size Pages
RAW 156 156 1216 26
UDP 150 150 1280 25
tw_sock_TCP 378 378 192 21
TCP 117 117 2368 13
fscache_cookie_jar 0 0 184 22
sgpool-128 68 90 5120 6
sgpool-64 72 72 2560 12
sgpool-32 175 175 1280 25
sgpool-16 275 275 640 25
blkdev_integrity 0 0 112 36
blkdev_queue 143 143 2808 11
blkdev_requests 346 346 376 21
fsnotify_event_holder 0 0 24 170
bip-256 7 7 4224 7
bip-128 0 0 2176 15
bip-16 0 0 384 21
sock_inode_cache 323 323 960 17
file_lock_cache 102 102 240 17
net_namespace 0 0 1984 16
shmem_inode_cache 1736 1736 1144 28
Acpi-State 306 306 80 51
Acpi-Namespace 1632 1632 40 102
task_delay_info 960 960 168 24
taskstats 144 144 328 24
proc_inode_cache 1440 1440 976 16
sigqueue 300 300 160 25
bdev_cache 144 144 1344 24
sysfs_dir_cache 22512 22512 144 28
filp 2134 2450 320 25
inode_cache 1260 1972 912 17
dentry 7006 11888 248 16
buffer_head 104786 107328 104 39
vm_area_struct 4131 4444 184 22
mm_struct 280 280 1152 28
files_cache 273 273 768 21
signal_cache 483 529 1408 23
sighand_cache 387 406 2240 14
task_struct 297 329 4208 7
anon_vma 4147 4536 144 28
shared_policy_node 510 510 48 85
numa_policy 3360 3360 72 56
radix_tree_node 5470 5740 568 28
idr_layer_cache 330 330 2112 15
dma-kmalloc-8192 0 0 8192 4
dma-kmalloc-4096 0 0 4096 8
dma-kmalloc-2048 0 0 2048 16
dma-kmalloc-1024 0 0 1024 16
dma-kmalloc-512 0 0 512 16
dma-kmalloc-256 0 0 256 16
dma-kmalloc-128 0 0 128 32
dma-kmalloc-64 0 0 64 64
dma-kmalloc-32 0 0 32 128
dma-kmalloc-16 0 0 16 256
dma-kmalloc-8 0 0 8 512
dma-kmalloc-192 0 0 192 21
dma-kmalloc-96 0 0 96 42
kmalloc-8192 28 28 8192 4
kmalloc-4096 909 928 4096 8
kmalloc-2048 646 704 2048 16
kmalloc-1024 1803 1808 1024 16
Cache Num Total Size Pages
kmalloc-512 1057 1152 512 16
kmalloc-256 4175 4304 256 16
kmalloc-128 2948 3456 128 32
kmalloc-64 13726 15040 64 64
kmalloc-32 1536 1536 32 128
kmalloc-16 4096 4096 16 256
kmalloc-8 7168 7168 8 512
kmalloc-192 68687 69111 192 21
kmalloc-96 3704 4032 96 42
kmem_cache_node 192 192 128 32
kmem_cache 160 160 256 16Attachment:
dmesg.txt Attachment:
xl-dmesg.txt _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |