[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] udev oops, and system boot failure, with 2.6.32.44 as PV guest



Jeremy Fitzhardinge writes ("Re: [Xen-devel] udev oops, and system boot 
failure, with 2.6.32.44 as PV guest"):
> On 08/18/2011 04:19 AM, Ian Jackson wrote:
> > This failure happens only on these two machines, for some reason.
> > I haven't tried 32-bit.

I see crashes with 32-on-64 too.

> At first glance it doesn't really look very Xen-related; alloc_fd isn't
> generally a place where anything Xen-specific happens.   Can you decode
> that to a specific line of code?

There doesn't seem to be much point, given that different crashes have
different locations.  I tried a number of boots and got the stack
backtraces you can see below.

Something is obviously completely buggered.

> I'm wondering if the access to "/sys/devices/virtual/bdi/1:13/uevent" is
> pertinent though; it could be one of our drivers which is doing the
> wrong thing which causes alloc_fd to explode.

No, it gives a different access each time.

> Is this expected, or does it indicate something wrong with your
> (initramfs?) confg?

I don't think anything is wrong with my initramfs.  It works fine with
other kernels :-).  The messages about volume group "rice-weevil"
being missing are simply because I reuse the host's initramfs, which
has had stuff about the host's disk layout encoded into it by the
host's initramfs-tools, and is harmless.

Ian.


Starting the hotplug events dispatcher: udevd[    1.240492] udev[835]: starting 
version 164
.
[    1.335497] BUG: unable to handle kernel NULL pointer dereference at (null)
[    1.335536] IP: [<c1051d38>] __wake_up_common+0x17/0x5c
[    1.335562] *pdpt = 000000000175c007 *pde = 0000000000000000 
[    1.335590] Oops: 0000 [#1] SMP 
[    1.335614] last sysfs file: /sys/kernel/uevent_seqnum
[    1.335627] Modules linked in: [last unloaded: scsi_wait_scan]
[    1.335653] 
[    1.335664] Pid: 844, comm: mv Not tainted (2.6.32.45 #1) 
[    1.335678] EIP: 0061:[<c1051d38>] EFLAGS: 00010093 CPU: 0
[    1.335692] EIP is at __wake_up_common+0x17/0x5c
[    1.335705] EAX: dfcb290c EBX: fffffff4 ECX: 00000001 EDX: 00000001
[    1.335720] ESI: dfcb0008 EDI: 00000001 EBP: dbb51e90 ESP: dbb51e78
[    1.335734]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[    1.335748] Process mv (pid: 844, ti=dbb50000 task=c4ef8f30 task.ti=dbb50000)
[    1.335763] Stack:
[    1.335772]  00000011 dbb51e84 00000001 dfcb2908 dfcb0008 00000011 dbb51eb0 
c1056e39
[    1.335828] <0> 00000001 c4ef8f30 00000001 00000001 c4ef8f30 00000011 
dbb51ebc c1062759
[    1.335889] <0> c4ef8f30 dbb51f5c c106f763 c4efef40 c4ef0044 c4eff444 
00000011 00000000
[    1.335958] Call Trace:
[    1.335976]  [<c1056e39>] ? __wake_up_sync_key+0x33/0x45
[    1.335996]  [<c1062759>] ? __wake_up_parent+0x1e/0x21
[    1.336015]  [<c106f763>] ? do_notify_parent+0x17e/0x19c
[    1.336036]  [<c10b880a>] ? perf_event_exit_task+0x1e/0x2b2
[    1.336059]  [<c146412e>] ? _write_lock_irq+0x18/0x2a
[    1.336070]  [<c1069d9a>] ? exit_ptrace+0xa3/0x10d
[    1.336070]  [<c1079c11>] ? switch_task_namespaces+0xf/0x3a
[    1.336070]  [<c106467f>] ? do_exit+0x553/0x608
[    1.336070]  [<c10647bc>] ? do_group_exit+0x88/0xab
[    1.336070]  [<c10647f2>] ? sys_exit_group+0x13/0x17
[    1.336070]  [<c102ea49>] ? syscall_call+0x7/0xb
[    1.336070] Code: 89 e5 e8 9b ff ff ff 5d c3 55 8b 80 88 02 00 00 89 e5 5d 
c3 55 89 e5 57 89 d7 56 53 83 ec 0c 89 4d f0 8b 58 04 83 c0 04 83 eb 0c <8b> 73 
0c 89 45 e8 83 ee 0c eb 2a 8b 03 89 fa ff 75 0c 8b 4d 08 
[    1.336070] EIP: [<c1051d38>] __wake_up_common+0x17/0x5c SS:ESP 0069:dbb51e78
[    1.336070] CR2: 0000000000000000
[    1.336070] ---[ end trace 59579aaa0506cac8 ]---
[    1.336070] Fixing recursive fault but reboot is needed!


Starting the hotplug events dispatcher: udevd[    1.200636] udev[839]: starting 
version 164
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...[    1.546234] BUG: unable to handle 
kernel NULL pointer dereference at 00000008
[    1.546258] IP: [<c11c1c01>] rb_erase+0x72/0x208
[    1.546272] *pdpt = 000000001fdcc007 *pde = 0000000000000000 
[    1.546284] Oops: 0002 [#1] SMP 
[    1.546295] last sysfs file: /sys/devices/virtual/vtconsole/vtcon0/uevent
[    1.546302] Modules linked in: [last unloaded: scsi_wait_scan]
[    1.546314] 
[    1.546319] Pid: 855, comm: udevd Not tainted (2.6.32.45 #1) 
[    1.546325] EIP: 0061:[<c11c1c01>] EFLAGS: 00010046 CPU: 0
[    1.546332] EIP is at rb_erase+0x72/0x208
[    1.546337] EAX: dbbfd004 EBX: 00000000 ECX: c4da8b84 EDX: 00000000
[    1.546344] ESI: c5ff0388 EDI: 00000000 EBP: c52c5f04 ESP: c52c5eec
[    1.546350]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
[    1.546357] Process udevd (pid: 855, ti=c52c4000 task=dbbab7b0 
task.ti=c52c4000)
[    1.546363] Stack:
[    1.546368]  00000000 00000000 00000000 dbbfd004 c5ff0380 00000000 c52c5f18 
c1078bd5
[    1.546393] <0> dbbfd004 048a2000 c52c5f84 c52c5f30 c1078c28 00000001 
c5ff0380 dbbfd004
[    1.546420] <0> ffffffff c52c5f44 c10793a4 c5ff0050 dbbab7b0 c52c5f94 
c52c5f7c c1064ae1
[    1.546451] Call Trace:
[    1.546461]  [<c1078bd5>] ? __remove_hrtimer+0x64/0x6c
[    1.546469]  [<c1078c28>] ? remove_hrtimer+0x4b/0x58
[    1.546478]  [<c10793a4>] ? hrtimer_try_to_cancel+0x24/0x3a
[    1.546488]  [<c1064ae1>] ? do_setitimer+0xaa/0x1f5
[    1.546497]  [<c10e93af>] ? __fput+0x161/0x169
[    1.546505]  [<c1064cd2>] ? alarm_setitimer+0x35/0x54
[    1.546515]  [<c106d1f6>] ? sys_alarm+0xb/0xd
[    1.546524]  [<c102ea49>] ? syscall_call+0x7/0xb
[    1.546530] Code: 8b 19 8b 51 04 89 5d ec 83 e3 fc 39 c3 89 5d f0 89 5d e8 
75 05 89 4d e8 eb 26 85 d2 74 0a 8b 3a 83 e7 03 0b 7d f0 89 3a 8b 7d f0 <89> 57 
08 8b 78 04 89 79 04 8b 58 04 8b 3b 83 e7 03 09 cf 89 3b 
[    1.546713] EIP: [<c11c1c01>] rb_erase+0x72/0x208 SS:ESP 0069:c52c5eec
[    1.546726] CR2: 0000000000000008
[    1.546733] ---[ end trace cafed11e7d7abcb5 ]---




Starting the hotplug events dispatcher: udevd[    1.260149] udev[838]: starting 
version 164
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...[    1.647871] BUG: unable to handle 
kernel NULL pointer dereference at (null)
[    1.647896] IP: [<c1051d38>] __wake_up_common+0x17/0x5c
[    1.647909] *pdpt = 000000000175c007 *pde = 0000000000000000 
[    1.647922] Oops: 0000 [#1] SMP 
[    1.647933] last sysfs file: /sys/devices/virtual/input/input0/mouse0/uevent
[    1.647940] Modules linked in: [last unloaded: scsi_wait_scan]
[    1.647952] 
[    1.647957] Pid: 934, comm: grep Not tainted (2.6.32.45 #1) 
[    1.647964] EIP: 0061:[<c1051d38>] EFLAGS: 00010093 CPU: 0
[    1.647971] EIP is at __wake_up_common+0x17/0x5c
[    1.647977] EAX: dea9f20c EBX: fffffff4 ECX: 00000001 EDX: 00000001
[    1.647983] ESI: dea90008 EDI: 00000001 EBP: dead5e90 ESP: dead5e78
[    1.647990]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[    1.647996] Process grep (pid: 934, ti=dead4000 task=dea98000 
task.ti=dead4000)
[    1.648003] Stack:
[    1.648008]  00000011 dead5e84 00000001 dea9f208 dea90008 00000011 dead5eb0 
c1056e39
[    1.648033] <0> 00000001 dea98000 00000001 00000001 dea98000 00000011 
dead5ebc c1062759
[    1.648053] <0> dea98000 dead5f5c c106f763 c4df39c0 c4df00c4 c4df3ec4 
00000011 00000000
[    1.648053] Call Trace:
[    1.648053]  [<c1056e39>] ? __wake_up_sync_key+0x33/0x45
[    1.648053]  [<c1062759>] ? __wake_up_parent+0x1e/0x21
[    1.648053]  [<c106f763>] ? do_notify_parent+0x17e/0x19c
[    1.648053]  [<c10b880a>] ? perf_event_exit_task+0x1e/0x2b2
[    1.648053]  [<c146412e>] ? _write_lock_irq+0x18/0x2a
[    1.648053]  [<c1069d9a>] ? exit_ptrace+0xa3/0x10d
[    1.648053]  [<c1079c11>] ? switch_task_namespaces+0xf/0x3a
[    1.648053]  [<c106467f>] ? do_exit+0x553/0x608
[    1.648053]  [<c10647bc>] ? do_group_exit+0x88/0xab
[    1.648053]  [<c10647f2>] ? sys_exit_group+0x13/0x17
[    1.648053]  [<c102ea49>] ? syscall_call+0x7/0xb
[    1.648053] Code: 89 e5 e8 9b ff ff ff 5d c3 55 8b 80 88 02 00 00 89 e5 5d 
c3 55 89 e5 57 89 d7 56 53 83 ec 0c 89 4d f0 8b 58 04 83 c0 04 83 eb 0c <8b> 73 
0c 89 45 e8 83 ee 0c eb 2a 8b 03 89 fa ff 75 0c 8b 4d 08 
[    1.648053] EIP: [<c1051d38>] __wake_up_common+0x17/0x5c SS:ESP 0069:dead5e78
[    1.648053] CR2: 0000000000000000
[    1.648053] ---[ end trace 6942a97668899ff4 ]---
[    1.648053] Fixing recursive fault but reboot is needed!



Using makefile-style concurrent boot in runlevel S.
[    1.133364] BUG: unable to handle kernel NULL pointer dereference at 00000004
[    1.133398] IP: [<c10767b5>] add_wait_queue+0x1b/0x36
[    1.133423] *pdpt = 000000001bfe4027 *pde = 0000000000000000 
[    1.133449] Oops: 0002 [#1] SMP 
[    1.133472] last sysfs file: /sys/kernel/uevent_seqnum
[    1.133485] Modules linked in: [last unloaded: scsi_wait_scan]
[    1.133510] 
[    1.133522] Pid: 808, comm: startpar Not tainted (2.6.32.45 #1) 
[    1.133536] EIP: 0061:[<c10767b5>] EFLAGS: 00010096 CPU: 0
[    1.133550] EIP is at add_wait_queue+0x1b/0x36
[    1.133563] EAX: c4f30208 EBX: c4f34908 ECX: dfce9f7c EDX: c4f3490c
[    1.133577] ESI: dfce9f70 EDI: 00000000 EBP: dfce9f20 ESP: dfce9f14
[    1.133591]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
[    1.133604] Process startpar (pid: 808, ti=dfce8000 task=c4f28a20 
task.ti=dfce8000)
[    1.133619] Stack:
[    1.133629]  dfce9f58 00000000 00000000 dfce9f4c c1063dd3 c5384380 c5384980 
dfce9f70
[    1.133683] <0> dfce9f48 00000010 c4f28a20 ffffffff 00000000 00000000 
dfce9f94 c1063fcf
[    1.133744] <0> c10e82b9 00000003 00000007 00000000 00000000 bf87ee9c 
00000000 00000000
[    1.133811] Call Trace:
[    1.133829]  [<c1063dd3>] ? do_wait+0x61/0x1d5
[    1.133847]  [<c1063fcf>] ? sys_wait4+0x88/0xa1
[    1.133865]  [<c10e82b9>] ? rw_verify_area+0x98/0xbb
[    1.133884]  [<c10626dc>] ? child_wait_callback+0x0/0x5f
[    1.133902]  [<c1063ffb>] ? sys_waitpid+0x13/0x15
[    1.133922]  [<c102ea49>] ? syscall_call+0x7/0xb
[    1.133934] Code: 89 39 89 c2 89 d8 e8 00 d9 3e 00 5b 5e 5f 5d c3 55 89 e5 
57 56 89 d6 53 89 c3 83 22 fe e8 70 da 3e 00 8b 7b 0
4 8d 4e 0c 8d 53 04 <89> 4f 04 89 7e 0c 89 56 10 89 c2 89 d8 89 4b 04 e8 cb d8 
3e 00 
[    1.134058] EIP: [<c10767b5>] add_wait_queue+0x1b/0x36 SS:ESP 0069:dfce9f14
[    1.134058] CR2: 0000000000000004
[    1.134058] ---[ end trace 85d46112ef8f4b48 ]---
[    1.134895] ------------[ cut here ]------------
[    1.134912] kernel BUG at kernel/exit.c:84!
[    1.134924] invalid opcode: 0000 [#2] SMP 
[    1.134948] last sysfs file: /sys/kernel/uevent_seqnum
[    1.134960] Modules linked in: [last unloaded: scsi_wait_scan]
[    1.134984] 
[    1.134996] Pid: 805, comm: rc Tainted: G      D    (2.6.32.45 #1) 
[    1.135010] EIP: 0061:[<c1062ec1>] EFLAGS: 00010046 CPU: 0
[    1.135025] EIP is at release_task+0x73/0x3d4
[    1.135038] EAX: 00000000 EBX: c4f28a20 ECX: c1668980 EDX: 02218e31
[    1.135054] ESI: c4f34900 EDI: c174e2e0 EBP: dfde7ed0 ESP: dfde7eb8
[    1.135068]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
[    1.135081] Process rc (pid: 805, ti=dfde6000 task=c4f29440 task.ti=dfde6000)
[    1.135096] Stack:
[    1.135105]  c4f28a20 dfde7ec4 c102ce90 bffffffd 00000328 c4f28a20 dfde7f38 
c1063967
[    1.135159] <0> 00003493 00000000 00000166 00000000 00000000 00000328 
00000001 00000000
[    1.135220] <0> 00000022 00000000 00000018 00000000 00000000 00000000 
00000000 00000000
[    1.135287] Call Trace:
[    1.135303]  [<c102ce90>] ? xen_spin_lock+0xa/0xe
[    1.135321]  [<c1063967>] ? wait_consider_task+0x745/0xb50
[    1.135340]  [<c1063e47>] ? do_wait+0xd5/0x1d5
[    1.135358]  [<c1063fcf>] ? sys_wait4+0x88/0xa1
[    1.135376]  [<c10626dc>] ? child_wait_callback+0x0/0x5f
[    1.135395]  [<c102ea49>] ? syscall_call+0x7/0xb
[    1.135405] Code: e8 d9 6c 00 00 8d 83 34 02 00 00 39 83 34 02 00 00 74 04 
0f 0b eb fe 8b b3 a8 03 00 00 85 f6 75 04 0f 0b eb f
e 8b 06 85 c0 75 04 <0f> 0b eb fe 8b 83 ac 03 00 00 89 45 f0 05 04 05 00 00 e8 
00 12 
[    1.135405] EIP: [<c1062ec1>] release_task+0x73/0x3d4 SS:ESP 0069:dfde7eb8
[    1.135405] ---[ end trace 85d46112ef8f4b49 ]---

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.