Xen project Mailing List

Re: [Xen-devel] udev oops, and system boot failure, with 2.6.32.44 as PV guest

To: Jeremy Fitzhardinge <jeremy@xxxxxxxx>

From: Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>

Date: Thu, 1 Sep 2011 16:35:51 +0100

Delivery-date: Thu, 01 Sep 2011 08:37:30 -0700

List-id: Xen developer discussion <xen-devel.lists.xensource.com>

Jeremy Fitzhardinge writes ("Re: [Xen-devel] udev oops, and system boot failure, with 2.6.32.44 as PV guest"): > On 08/18/2011 04:19 AM, Ian Jackson wrote: > > This failure happens only on these two machines, for some reason. > > I haven't tried 32-bit. I see crashes with 32-on-64 too. > At first glance it doesn't really look very Xen-related; alloc_fd isn't > generally a place where anything Xen-specific happens. Can you decode > that to a specific line of code? There doesn't seem to be much point, given that different crashes have different locations. I tried a number of boots and got the stack backtraces you can see below. Something is obviously completely buggered. > I'm wondering if the access to "/sys/devices/virtual/bdi/1:13/uevent" is > pertinent though; it could be one of our drivers which is doing the > wrong thing which causes alloc_fd to explode. No, it gives a different access each time. > Is this expected, or does it indicate something wrong with your > (initramfs?) confg? I don't think anything is wrong with my initramfs. It works fine with other kernels :-). The messages about volume group "rice-weevil" being missing are simply because I reuse the host's initramfs, which has had stuff about the host's disk layout encoded into it by the host's initramfs-tools, and is harmless. Ian. Starting the hotplug events dispatcher: udevd[ 1.240492] udev[835]: starting version 164 . [ 1.335497] BUG: unable to handle kernel NULL pointer dereference at (null) [ 1.335536] IP: [<c1051d38>] __wake_up_common+0x17/0x5c [ 1.335562] *pdpt = 000000000175c007 *pde = 0000000000000000 [ 1.335590] Oops: 0000 [#1] SMP [ 1.335614] last sysfs file: /sys/kernel/uevent_seqnum [ 1.335627] Modules linked in: [last unloaded: scsi_wait_scan] [ 1.335653] [ 1.335664] Pid: 844, comm: mv Not tainted (2.6.32.45 #1) [ 1.335678] EIP: 0061:[<c1051d38>] EFLAGS: 00010093 CPU: 0 [ 1.335692] EIP is at __wake_up_common+0x17/0x5c [ 1.335705] EAX: dfcb290c EBX: fffffff4 ECX: 00000001 EDX: 00000001 [ 1.335720] ESI: dfcb0008 EDI: 00000001 EBP: dbb51e90 ESP: dbb51e78 [ 1.335734] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069 [ 1.335748] Process mv (pid: 844, ti=dbb50000 task=c4ef8f30 task.ti=dbb50000) [ 1.335763] Stack: [ 1.335772] 00000011 dbb51e84 00000001 dfcb2908 dfcb0008 00000011 dbb51eb0 c1056e39 [ 1.335828] <0> 00000001 c4ef8f30 00000001 00000001 c4ef8f30 00000011 dbb51ebc c1062759 [ 1.335889] <0> c4ef8f30 dbb51f5c c106f763 c4efef40 c4ef0044 c4eff444 00000011 00000000 [ 1.335958] Call Trace: [ 1.335976] [<c1056e39>] ? __wake_up_sync_key+0x33/0x45 [ 1.335996] [<c1062759>] ? __wake_up_parent+0x1e/0x21 [ 1.336015] [<c106f763>] ? do_notify_parent+0x17e/0x19c [ 1.336036] [<c10b880a>] ? perf_event_exit_task+0x1e/0x2b2 [ 1.336059] [<c146412e>] ? _write_lock_irq+0x18/0x2a [ 1.336070] [<c1069d9a>] ? exit_ptrace+0xa3/0x10d [ 1.336070] [<c1079c11>] ? switch_task_namespaces+0xf/0x3a [ 1.336070] [<c106467f>] ? do_exit+0x553/0x608 [ 1.336070] [<c10647bc>] ? do_group_exit+0x88/0xab [ 1.336070] [<c10647f2>] ? sys_exit_group+0x13/0x17 [ 1.336070] [<c102ea49>] ? syscall_call+0x7/0xb [ 1.336070] Code: 89 e5 e8 9b ff ff ff 5d c3 55 8b 80 88 02 00 00 89 e5 5d c3 55 89 e5 57 89 d7 56 53 83 ec 0c 89 4d f0 8b 58 04 83 c0 04 83 eb 0c <8b> 73 0c 89 45 e8 83 ee 0c eb 2a 8b 03 89 fa ff 75 0c 8b 4d 08 [ 1.336070] EIP: [<c1051d38>] __wake_up_common+0x17/0x5c SS:ESP 0069:dbb51e78 [ 1.336070] CR2: 0000000000000000 [ 1.336070] ---[ end trace 59579aaa0506cac8 ]--- [ 1.336070] Fixing recursive fault but reboot is needed! Starting the hotplug events dispatcher: udevd[ 1.200636] udev[839]: starting version 164 . Synthesizing the initial hotplug events...done. Waiting for /dev to be fully populated...[ 1.546234] BUG: unable to handle kernel NULL pointer dereference at 00000008 [ 1.546258] IP: [<c11c1c01>] rb_erase+0x72/0x208 [ 1.546272] *pdpt = 000000001fdcc007 *pde = 0000000000000000 [ 1.546284] Oops: 0002 [#1] SMP [ 1.546295] last sysfs file: /sys/devices/virtual/vtconsole/vtcon0/uevent [ 1.546302] Modules linked in: [last unloaded: scsi_wait_scan] [ 1.546314] [ 1.546319] Pid: 855, comm: udevd Not tainted (2.6.32.45 #1) [ 1.546325] EIP: 0061:[<c11c1c01>] EFLAGS: 00010046 CPU: 0 [ 1.546332] EIP is at rb_erase+0x72/0x208 [ 1.546337] EAX: dbbfd004 EBX: 00000000 ECX: c4da8b84 EDX: 00000000 [ 1.546344] ESI: c5ff0388 EDI: 00000000 EBP: c52c5f04 ESP: c52c5eec [ 1.546350] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069 [ 1.546357] Process udevd (pid: 855, ti=c52c4000 task=dbbab7b0 task.ti=c52c4000) [ 1.546363] Stack: [ 1.546368] 00000000 00000000 00000000 dbbfd004 c5ff0380 00000000 c52c5f18 c1078bd5 [ 1.546393] <0> dbbfd004 048a2000 c52c5f84 c52c5f30 c1078c28 00000001 c5ff0380 dbbfd004 [ 1.546420] <0> ffffffff c52c5f44 c10793a4 c5ff0050 dbbab7b0 c52c5f94 c52c5f7c c1064ae1 [ 1.546451] Call Trace: [ 1.546461] [<c1078bd5>] ? __remove_hrtimer+0x64/0x6c [ 1.546469] [<c1078c28>] ? remove_hrtimer+0x4b/0x58 [ 1.546478] [<c10793a4>] ? hrtimer_try_to_cancel+0x24/0x3a [ 1.546488] [<c1064ae1>] ? do_setitimer+0xaa/0x1f5 [ 1.546497] [<c10e93af>] ? __fput+0x161/0x169 [ 1.546505] [<c1064cd2>] ? alarm_setitimer+0x35/0x54 [ 1.546515] [<c106d1f6>] ? sys_alarm+0xb/0xd [ 1.546524] [<c102ea49>] ? syscall_call+0x7/0xb [ 1.546530] Code: 8b 19 8b 51 04 89 5d ec 83 e3 fc 39 c3 89 5d f0 89 5d e8 75 05 89 4d e8 eb 26 85 d2 74 0a 8b 3a 83 e7 03 0b 7d f0 89 3a 8b 7d f0 <89> 57 08 8b 78 04 89 79 04 8b 58 04 8b 3b 83 e7 03 09 cf 89 3b [ 1.546713] EIP: [<c11c1c01>] rb_erase+0x72/0x208 SS:ESP 0069:c52c5eec [ 1.546726] CR2: 0000000000000008 [ 1.546733] ---[ end trace cafed11e7d7abcb5 ]--- Starting the hotplug events dispatcher: udevd[ 1.260149] udev[838]: starting version 164 . Synthesizing the initial hotplug events...done. Waiting for /dev to be fully populated...[ 1.647871] BUG: unable to handle kernel NULL pointer dereference at (null) [ 1.647896] IP: [<c1051d38>] __wake_up_common+0x17/0x5c [ 1.647909] *pdpt = 000000000175c007 *pde = 0000000000000000 [ 1.647922] Oops: 0000 [#1] SMP [ 1.647933] last sysfs file: /sys/devices/virtual/input/input0/mouse0/uevent [ 1.647940] Modules linked in: [last unloaded: scsi_wait_scan] [ 1.647952] [ 1.647957] Pid: 934, comm: grep Not tainted (2.6.32.45 #1) [ 1.647964] EIP: 0061:[<c1051d38>] EFLAGS: 00010093 CPU: 0 [ 1.647971] EIP is at __wake_up_common+0x17/0x5c [ 1.647977] EAX: dea9f20c EBX: fffffff4 ECX: 00000001 EDX: 00000001 [ 1.647983] ESI: dea90008 EDI: 00000001 EBP: dead5e90 ESP: dead5e78 [ 1.647990] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069 [ 1.647996] Process grep (pid: 934, ti=dead4000 task=dea98000 task.ti=dead4000) [ 1.648003] Stack: [ 1.648008] 00000011 dead5e84 00000001 dea9f208 dea90008 00000011 dead5eb0 c1056e39 [ 1.648033] <0> 00000001 dea98000 00000001 00000001 dea98000 00000011 dead5ebc c1062759 [ 1.648053] <0> dea98000 dead5f5c c106f763 c4df39c0 c4df00c4 c4df3ec4 00000011 00000000 [ 1.648053] Call Trace: [ 1.648053] [<c1056e39>] ? __wake_up_sync_key+0x33/0x45 [ 1.648053] [<c1062759>] ? __wake_up_parent+0x1e/0x21 [ 1.648053] [<c106f763>] ? do_notify_parent+0x17e/0x19c [ 1.648053] [<c10b880a>] ? perf_event_exit_task+0x1e/0x2b2 [ 1.648053] [<c146412e>] ? _write_lock_irq+0x18/0x2a [ 1.648053] [<c1069d9a>] ? exit_ptrace+0xa3/0x10d [ 1.648053] [<c1079c11>] ? switch_task_namespaces+0xf/0x3a [ 1.648053] [<c106467f>] ? do_exit+0x553/0x608 [ 1.648053] [<c10647bc>] ? do_group_exit+0x88/0xab [ 1.648053] [<c10647f2>] ? sys_exit_group+0x13/0x17 [ 1.648053] [<c102ea49>] ? syscall_call+0x7/0xb [ 1.648053] Code: 89 e5 e8 9b ff ff ff 5d c3 55 8b 80 88 02 00 00 89 e5 5d c3 55 89 e5 57 89 d7 56 53 83 ec 0c 89 4d f0 8b 58 04 83 c0 04 83 eb 0c <8b> 73 0c 89 45 e8 83 ee 0c eb 2a 8b 03 89 fa ff 75 0c 8b 4d 08 [ 1.648053] EIP: [<c1051d38>] __wake_up_common+0x17/0x5c SS:ESP 0069:dead5e78 [ 1.648053] CR2: 0000000000000000 [ 1.648053] ---[ end trace 6942a97668899ff4 ]--- [ 1.648053] Fixing recursive fault but reboot is needed! Using makefile-style concurrent boot in runlevel S. [ 1.133364] BUG: unable to handle kernel NULL pointer dereference at 00000004 [ 1.133398] IP: [<c10767b5>] add_wait_queue+0x1b/0x36 [ 1.133423] *pdpt = 000000001bfe4027 *pde = 0000000000000000 [ 1.133449] Oops: 0002 [#1] SMP [ 1.133472] last sysfs file: /sys/kernel/uevent_seqnum [ 1.133485] Modules linked in: [last unloaded: scsi_wait_scan] [ 1.133510] [ 1.133522] Pid: 808, comm: startpar Not tainted (2.6.32.45 #1) [ 1.133536] EIP: 0061:[<c10767b5>] EFLAGS: 00010096 CPU: 0 [ 1.133550] EIP is at add_wait_queue+0x1b/0x36 [ 1.133563] EAX: c4f30208 EBX: c4f34908 ECX: dfce9f7c EDX: c4f3490c [ 1.133577] ESI: dfce9f70 EDI: 00000000 EBP: dfce9f20 ESP: dfce9f14 [ 1.133591] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069 [ 1.133604] Process startpar (pid: 808, ti=dfce8000 task=c4f28a20 task.ti=dfce8000) [ 1.133619] Stack: [ 1.133629] dfce9f58 00000000 00000000 dfce9f4c c1063dd3 c5384380 c5384980 dfce9f70 [ 1.133683] <0> dfce9f48 00000010 c4f28a20 ffffffff 00000000 00000000 dfce9f94 c1063fcf [ 1.133744] <0> c10e82b9 00000003 00000007 00000000 00000000 bf87ee9c 00000000 00000000 [ 1.133811] Call Trace: [ 1.133829] [<c1063dd3>] ? do_wait+0x61/0x1d5 [ 1.133847] [<c1063fcf>] ? sys_wait4+0x88/0xa1 [ 1.133865] [<c10e82b9>] ? rw_verify_area+0x98/0xbb [ 1.133884] [<c10626dc>] ? child_wait_callback+0x0/0x5f [ 1.133902] [<c1063ffb>] ? sys_waitpid+0x13/0x15 [ 1.133922] [<c102ea49>] ? syscall_call+0x7/0xb [ 1.133934] Code: 89 39 89 c2 89 d8 e8 00 d9 3e 00 5b 5e 5f 5d c3 55 89 e5 57 56 89 d6 53 89 c3 83 22 fe e8 70 da 3e 00 8b 7b 0 4 8d 4e 0c 8d 53 04 <89> 4f 04 89 7e 0c 89 56 10 89 c2 89 d8 89 4b 04 e8 cb d8 3e 00 [ 1.134058] EIP: [<c10767b5>] add_wait_queue+0x1b/0x36 SS:ESP 0069:dfce9f14 [ 1.134058] CR2: 0000000000000004 [ 1.134058] ---[ end trace 85d46112ef8f4b48 ]--- [ 1.134895] ------------[ cut here ]------------ [ 1.134912] kernel BUG at kernel/exit.c:84! [ 1.134924] invalid opcode: 0000 [#2] SMP [ 1.134948] last sysfs file: /sys/kernel/uevent_seqnum [ 1.134960] Modules linked in: [last unloaded: scsi_wait_scan] [ 1.134984] [ 1.134996] Pid: 805, comm: rc Tainted: G D (2.6.32.45 #1) [ 1.135010] EIP: 0061:[<c1062ec1>] EFLAGS: 00010046 CPU: 0 [ 1.135025] EIP is at release_task+0x73/0x3d4 [ 1.135038] EAX: 00000000 EBX: c4f28a20 ECX: c1668980 EDX: 02218e31 [ 1.135054] ESI: c4f34900 EDI: c174e2e0 EBP: dfde7ed0 ESP: dfde7eb8 [ 1.135068] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069 [ 1.135081] Process rc (pid: 805, ti=dfde6000 task=c4f29440 task.ti=dfde6000) [ 1.135096] Stack: [ 1.135105] c4f28a20 dfde7ec4 c102ce90 bffffffd 00000328 c4f28a20 dfde7f38 c1063967 [ 1.135159] <0> 00003493 00000000 00000166 00000000 00000000 00000328 00000001 00000000 [ 1.135220] <0> 00000022 00000000 00000018 00000000 00000000 00000000 00000000 00000000 [ 1.135287] Call Trace: [ 1.135303] [<c102ce90>] ? xen_spin_lock+0xa/0xe [ 1.135321] [<c1063967>] ? wait_consider_task+0x745/0xb50 [ 1.135340] [<c1063e47>] ? do_wait+0xd5/0x1d5 [ 1.135358] [<c1063fcf>] ? sys_wait4+0x88/0xa1 [ 1.135376] [<c10626dc>] ? child_wait_callback+0x0/0x5f [ 1.135395] [<c102ea49>] ? syscall_call+0x7/0xb [ 1.135405] Code: e8 d9 6c 00 00 8d 83 34 02 00 00 39 83 34 02 00 00 74 04 0f 0b eb fe 8b b3 a8 03 00 00 85 f6 75 04 0f 0b eb f e 8b 06 85 c0 75 04 <0f> 0b eb fe 8b 83 ac 03 00 00 89 45 f0 05 04 05 00 00 e8 00 12 [ 1.135405] EIP: [<c1062ec1>] release_task+0x73/0x3d4 SS:ESP 0069:dfde7eb8 [ 1.135405] ---[ end trace 85d46112ef8f4b49 ]--- _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.