Re: [Xen-devel] Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping

[Sander Eikelenboom <linux@xxxxxxxxxxxxxx>]

Sorry, something went wrong with copy and pasting, here is the complete output:

(gdb) tar rem :9999
Remote debugging using :9999
[New Thread 0]
[Switching to Thread 0]
smp_call_function_many (mask=0xffffffff81ce8e70, func=0xffffffff81007c4d 
<stop_self>, info=0x0, wait=true) at kernel/smp.c:104
104             while (data->flags & CSD_FLAG_LOCK)
(gdb) thread apply all bt
[New Thread 1]
[New Thread 2]
[New Thread 3]

Thread 4 (Thread 3):
#0  0xffffffff8100130a in hypercall_page ()
#1  0xffffffffffff02dd in ?? ()
#2  0x0000000000000080 in irq_stack_union ()
#3  0xffffffff81007c9a in stop_self (v=<value optimized out>) at 
/usr/src/new/linux-2.6-usb-msi/arch/x86/include/asm/xen/hypercall.h:413
#4  0xffffffff81077adf in generic_smp_call_function_interrupt () at 
kernel/smp.c:200
#5  0xffffffff81007eb2 in xen_call_function_interrupt (irq=<value optimized 
out>, dev_id=<value optimized out>) at arch/x86/xen/smp.c:472
#6  0xffffffff8109907b in handle_IRQ_event (irq=286, action=0xffff88001e8257e0) 
at kernel/irq/handle.c:68
#7  0xffffffff8109ae9c in handle_percpu_irq (irq=286, desc=0xffff88001ea05500) 
at kernel/irq/chip.c:684
#8  0xffffffff8127a3b0 in __xen_evtchn_do_upcall () at 
include/linux/irqdesc.h:119
#9  0xffffffff8127ab80 in xen_evtchn_do_upcall (regs=<value optimized out>) at 
drivers/xen/events.c:1143
#10 0xffffffff8100bc2e in xen_do_hypervisor_callback () at 
arch/x86/kernel/entry_64.S:1233
#11 0xffff88001e9e3e28 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 3 (Thread 2):
#0  0xffffffff8100130a in hypercall_page ()
#1  0xffffffff81006fbd in xen_force_evtchn_callback () at 
/usr/src/new/linux-2.6-usb-msi/arch/x86/include/asm/xen/hypercall.h:362
#2  0xffffffff81077adf in generic_smp_call_function_interrupt () at 
kernel/smp.c:200
#3  0xffffffff81007eb2 in xen_call_function_interrupt (irq=<value optimized 
out>, dev_id=<value optimized out>) at arch/x86/xen/smp.c:472
#4  0xffffffff8109907b in handle_IRQ_event (irq=291, action=0xffff88001e825600) 
at kernel/irq/handle.c:68
#5  0xffffffff8109ae9c in handle_percpu_irq (irq=291, desc=0xffff88001ea05000) 
at kernel/irq/chip.c:684
#6  0xffffffff8127a3b0 in __xen_evtchn_do_upcall () at 
include/linux/irqdesc.h:119
#7  0xffffffff8127ab80 in xen_evtchn_do_upcall (regs=<value optimized out>) at 
drivers/xen/events.c:1143
#8  0xffffffff8100bc2e in xen_do_hypervisor_callback () at 
arch/x86/kernel/entry_64.S:1233
#9  0xffff88001e9e1e28 in ?? ()
#10 0x0000000000000000 in ?? ()
Current language:  auto; currently asm

Thread 2 (Thread 1):
#0  0xffffffff8100130a in hypercall_page ()
#1  0xffffffff81ce8e70 in cpu_possible_bits ()
#2  0x0000000000000080 in irq_stack_union ()
#3  0xffffffff81007c9a in stop_self (v=<value optimized out>) at 
/usr/src/new/linux-2.6-usb-msi/arch/x86/include/asm/xen/hypercall.h:413
#4  0xffffffff81077adf in generic_smp_call_function_interrupt () at 
kernel/smp.c:200
#5  0xffffffff81007eb2 in xen_call_function_interrupt (irq=<value optimized 
out>, dev_id=<value optimized out>) at arch/x86/xen/smp.c:472
#6  0xffffffff8109907b in handle_IRQ_event (irq=296, action=0xffff88001e825420) 
at kernel/irq/handle.c:68
#7  0xffffffff8109ae9c in handle_percpu_irq (irq=296, desc=0xffff88001e824b00) 
at kernel/irq/chip.c:684
#8  0xffffffff8127a3b0 in __xen_evtchn_do_upcall () at 
include/linux/irqdesc.h:119
#9  0xffffffff8127ab80 in xen_evtchn_do_upcall (regs=<value optimized out>) at 
drivers/xen/events.c:1143
#10 0xffffffff8100bc2e in xen_do_hypervisor_callback () at 
arch/x86/kernel/entry_64.S:1233
#11 0xffff88001e9d7e28 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 1 (Thread 0):
#0  smp_call_function_many (mask=0xffffffff81ce8e70, func=0xffffffff81007c4d 
<stop_self>, info=0x0, wait=true) at kernel/smp.c:104
#1  0xffffffff81077a5a in smp_call_function (func=0x40 <irq_stack_union+64>, 
info=<value optimized out>, wait=<value optimized out>) at kernel/smp.c:506
#2  0xffffffff81007eda in xen_stop_other_cpus (wait=<value optimized out>) at 
arch/x86/xen/smp.c:431
#3  0xffffffff81003172 in xen_reboot () at 
/usr/src/new/linux-2.6-usb-msi/arch/x86/include/asm/smp.h:81
#4  0xffffffff810031b5 in xen_machine_halt () at arch/x86/xen/enlighten.c:1039
#5  0xffffffff81023795 in machine_halt () at arch/x86/kernel/reboot.c:726
#6  0xffffffff8105e9b3 in kernel_halt () at kernel/sys.c:336
#7  0xffffffff8105eb03 in sys_reboot (magic1=-18751827, magic2=672274793, 
cmd=3454992675, arg=0x0) at kernel/sys.c:407
#8  0xffffffff8100acc2 in system_call_fastpath () at 
arch/x86/kernel/entry_64.S:479
#9  0x0000000000000202 in irq_stack_union ()
#10 0x0000000000000000 in ?? ()
Current language:  auto; currently c


Friday, November 19, 2010, 6:07:43 PM, you wrote:

> On 11/18/2010 11:57 AM, Sander Eikelenboom wrote:
>> Hi Jeremy/Keir,
>>
>> I have tried to add the following lines:
>>        if (!strcmp(type, "console"))
>>                return 0;
>>
>> But it seems to be a red herring .. although the "the xenbus_dev_shutdown: 
>> device/console/0: Initialising != Connected, skipping" warning is gone.
>> The PV domU guest still doesn't get completely halted:
>> - It stays in a running state
>> - It uses 100% cpu according to xentop
>> - I can still connect to it's console
>>
>> So probably something is stuck in a waiting loop, which doesn't sleep in 
>> between and consumes 100% cpu.

> Hm, I generally see clean shutdowns of my multi-VCPU domains.

> Can you do

> # gdbsx -a <domid> <arch-size> 9999        (arch-size is 32 or 64)

> then

> $ gdb vmlinux
> (gdb) tar rem :9999
> (gdb) thread apply all bt

> to see what's going on?  (You may need to enable CONFIG_DEBUG_INFO to
> get useful info if you haven't already.)

>     J


>> Last lines of domU's console(with some additional printk's in 
>> xenbus_dev_shutdown):
>>
>> Cannot access the Hardware Clock via any known method.
>> Use the --debug option to see the details of our search for an access method.
>> Stopping enhanced syslogd: rsyslogd.
>> Asking all remaining processes to terminate...done.
>> All processes ended within 2 seconds....done.
>> Deconfiguring network interfaces...done.
>> Cleaning up ifupdown....
>> Deactivating swap...done.
>> Unmounting local filesystems...done.
>> Will now halt.
>> [   46.643035] md: stopping all md devices.
>> [   47.643320] xenbus_dev_shutdown:  trying shutdown of device/vif/0: 
>> Connected
>> [   47.716448] xenbus_dev_shutdown: result of shutdown of device/vif/0: 
>> Closed
>> [   47.716461] xenbus_dev_shutdown:  trying shutdown of device/vbd/51714: 
>> Connected
>> [   47.772293] xenbus_dev_shutdown: result of shutdown of device/vbd/51714: 
>> Closed
>> [   47.772306] xenbus_dev_shutdown:  trying shutdown of device/vbd/51713: 
>> Connected
>> [   47.829384] xenbus_dev_shutdown: result of shutdown of device/vbd/51713: 
>> Closed
>> [   47.829415] System halted.
>>
>>
>>
>> it's a domU using a 2.6.37-rc2 kernel, file: based disk access, nothing very 
>> special.
>>
>> the only difference that triggers it is:
>> vcpus=1 works fine ..
>> vcpus>1 symptoms above ..
>>
>> I have also added xend.log, with first a boot and shutdown with vpcus=1 and 
>> then a boot and shutdown with vpcus=4
>> I left the domain(with 4vcpus) running with 100% cpu for 10 minutes, but 
>> doesn't seem to hit any timeout, and keeps running.
>> On 20:40 it ends up stuck, on 20:51 i'm shooting the domain off with xm 
>> destroy.
>>
>>
>> Any pointers for adding extra debug info ?
>>
>> --
>> Sander
>>
>>
>>
>>
>> Wednesday, November 17, 2010, 10:57:47 PM, you wrote:
>>
>>> On 11/17/2010 12:22 PM, Sander Eikelenboom wrote:
>>>> Ah yes, 2.6.18 does contain the following line  in xenbus_probe.c and 
>>>> 2.6.37-rc2 not:
>>>>
>>>> -       if (!strcmp(type, "console"))
>>>> -               return 0;
>>>> It seems to be changed,
>>>>
>>>>
>>>> There seems to have been a patch 
>>>> http://lists.xensource.com/archives/html/xen-devel/2009-11/msg01072.html 
>>>> but it seems it's not applied to xenbus in linux upstream.
>>> And does this patch fix things for you?
>>> But I have to say that's pretty gross.  Is it really the right thing to do?
>>>     J
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>>> http://lists.xensource.com/xen-devel




-- 
Best regards,
 Sander                            mailto:linux@xxxxxxxxxxxxxx


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel