[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] changeset: 6884 update


  • To: Xen Devs <xen-devel@xxxxxxxxxxxxxxxxxxx>
  • From: Ted Kaczmarek <tedkaz@xxxxxxxxx>
  • Date: Fri, 16 Sep 2005 08:42:31 -0400
  • Delivery-date: Fri, 16 Sep 2005 12:40:17 +0000
  • Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:subject:from:to:content-type:date:message-id:mime-version:x-mailer:content-transfer-encoding; b=uQq7GaHlhajiWFOl6/8Nd0sJEpLcyfQglLe0PXJVtvcIH/ZkTestcAVeeuRZVGrRR90OOeXFpcComXQp/aRG4Ohue+hlIbmEF2T+DXeofgRBLCD+PXvVO17hG5NOrN/Bq8sUQIeduEOI7tgCs86PwkGYT9OqecdAFbSKkI4HQ0k=
  • List-id: Xen developer discussion <xen-devel.lists.xensource.com>

FC4, Tyan 2462 SMP, 2 gig, most advanced networking modules built for
Dom0 and DomU.

On first attempt 
start xend manually

Create 1st DomU, it crashed, this is consistent even after reboot.


######################
DomU Kernel Crash CLI
######################

Freeing unused kernel memory: 148k freed
int3: 0000 [#1]
PREEMPT SMP
Modules linked in:
CPU:    1
EIP:    0061:[<c039a4bf>]    Not tainted VLI
EFLAGS: 00000246   (2.6.12-xenU)
EIP is at start_secondary+0x2f/0xd0
eax: 00000001   ebx: 00000000   ecx: 00000007   edx: c1540000
esi: 00000000   edi: 00000000   ebp: 00000000   esp: c1541fb8
ds: 007b   es: 007b   ss: 0069
Process swapper (pid: 0, threadinfo=c1540000 task=c009a530)
Stack: 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
       00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
       00000000 00000000
Call Trace:
Code: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc <cc> cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
<0>Kernel panic - not syncing: Attempted to kill the idle task!

###########
Log Snippet
###########

DEBUG (blkif:24) exception looking up device number for sda2: [Errno 2]
No such file or directory: '/dev/sda2'

Created second DomU with this one crashed, no problems.

Created first DomU, xm list now shows two instances

init 0 to second DomU, stop xend , restart xend , first DomU get created
with no problems.

Also some interesting messages

[2005-09-16 07:29:12 xend] DEBUG (blkif:24) exception looking up device
number for hda2: [Errno 2] No such file or directory: '/dev/hda2'
[2005-09-16 07:29:12 xend] INFO (XendRoot:141) EVENT>
xend.domain.unpause ['inyoureyes-domain2', 5]
[2005-09-16 07:29:25 xend] DEBUG (image:159) initDomain: cpu=-1
mem_kb=134144 ssidref=-1 dom=6
[2005-09-16 07:29:25 xend] DEBUG (XendDomainInfo:822) init_domain>
Created domain=6 name=inyoureyes-domain3 memory=131
[2005-09-16 07:29:25 xend] INFO (XendRoot:141) EVENT> xend.domain.create
['inyoureyes-domain3', 6]
[2005-09-16 07:29:25 xend] INFO (image:196) buildDomain os=linux dom=6
vcpus=2
[2005-09-16 07:29:25 xend] DEBUG (image:240) dom            = 6
[2005-09-16 07:29:25 xend] DEBUG (image:241) image
= /boot/vmlinuz-2.6.12-xenU
[2005-09-16 07:29:25 xend] DEBUG (image:242) store_evtchn   = 1
[2005-09-16 07:29:25 xend] DEBUG (image:243) console_evtchn = 2
[2005-09-16 07:29:25 xend] DEBUG (image:244) cmdline        =
root=/dev/hda3 ro selinux=0 3
[2005-09-16 07:29:25 xend] DEBUG (image:245) ramdisk        =
[2005-09-16 07:29:25 xend] DEBUG (image:246) flags          = 0
[2005-09-16 07:29:25 xend] DEBUG (image:247) vcpus          = 2
[2005-09-16 07:29:25 xend] DEBUG (blkif:24) exception looking up device
number for hda3: [Errno 2] No such file or directory: '/dev/hda3'
[2005-09-16 07:29:26 xend] INFO (XendRoot:141) EVENT>
xend.domain.unpause ['inyoureyes-domain3', 6]


inyoureyes-domain2 started fine

inyoureyes-domain3 got stuck at 
xen_net: Initialising virtual ethernet driver.
xen_net: Using grant tables.


For these tests I have most of the advanced networking modules being
built if that matters.

The good part is that xm destroy is able to kill inyoureyes-domain3.
>From what I have been seeing xm destroy problem is contained to crashed
and duped Domu's only, this appears to be consistent to what I read on
the list.

I restart inyoureyes-domain3 and xm list shows it as paused, issuing an
xm unpause it now shows blocked, but issuing xm console fails with 
xenconsole: Could not read tty from store: No such file or directory.
Than I issue xm list and inyoureyes-domain3 is no longer showing up.

Correlating log snippets

[2005-09-16 07:33:48 xend] DEBUG (XendDomain:472)
domain_restart_schedule> 6 halt 1
[2005-09-16 07:33:48 xend] INFO (XendDomain:485) Cancelling restart for
domain: name=inyoureyes-domain3 id=6
[2005-09-16 07:33:48 xend] INFO (XendDomain:514) Destroying domain:
name=inyoureyes-domain3
[2005-09-16 07:33:48 xend] INFO (XendRoot:141) EVENT>
xend.domain.destroy ['inyoureyes-domain3', 6]
[2005-09-16 07:33:48 xend] INFO (XendRoot:141) EVENT>
xend.domain.restart ['inyoureyes-domain3', 6, 'cancel']
[2005-09-16 07:33:48 xend] DEBUG (XendDomain:240) XendDomain>reap>
domain died name=?? id=1
[2005-09-16 07:33:48 xend] INFO (XendRoot:141) EVENT> xend.domain.exit
['??', 1, 'crash']
[2005-09-16 07:35:34 xend] DEBUG (image:159) initDomain: cpu=-1
mem_kb=134144 ssidref=-1 dom=7
[2005-09-16 07:35:34 xend] DEBUG (XendDomainInfo:822) init_domain>
Created domain=7 name=inyoureyes-domain3 memory=131
[2005-09-16 07:35:34 xend] INFO (XendRoot:141) EVENT> xend.domain.create
['inyoureyes-domain3', 7]
[2005-09-16 07:35:34 xend] INFO (image:196) buildDomain os=linux dom=7
vcpus=2
[2005-09-16 07:35:34 xend] DEBUG (image:240) dom            = 7
[2005-09-16 07:35:34 xend] DEBUG (image:241) image
= /boot/vmlinuz-2.6.12-xenU
[2005-09-16 07:35:34 xend] DEBUG (image:242) store_evtchn   = 1
[2005-09-16 07:35:34 xend] DEBUG (image:243) console_evtchn = 2
[2005-09-16 07:35:34 xend] DEBUG (image:244) cmdline        =
root=/dev/hda3 ro selinux=0 3
[2005-09-16 07:35:34 xend] DEBUG (image:245) ramdisk        =
[2005-09-16 07:35:34 xend] DEBUG (image:246) flags          = 0
[2005-09-16 07:35:34 xend] DEBUG (image:247) vcpus          = 2
[2005-09-16 07:35:34 xend] DEBUG (blkif:24) exception looking up device
number for hda3: [Errno 2] No such file or directory: '/dev/hda3'
[2005-09-16 07:35:35 xend] INFO (XendRoot:141) EVENT>
xend.domain.unpause ['inyoureyes-domain3', 6]
[2005-09-16 07:35:40 xend] INFO (XendRoot:141) EVENT> xend.domain.died
['inyoureyes-domain3', None]
[2005-09-16 07:35:53 xend] INFO (XendRoot:141) EVENT>
xend.domain.unpause ['inyoureyes-domain3', 7]
[2005-09-16 07:36:15 xend] DEBUG (XendDomain:240) XendDomain>reap>
domain died name=?? id=1
[2005-09-16 07:36:15 xend] DEBUG (XendDomain:240) XendDomain>reap>
domain died name=inyoureyes-domain3 id=7
[2005-09-16 07:36:15 xend] INFO (XendRoot:141) EVENT> xend.domain.exit
['??', 1, 'crash']
[2005-09-16 07:36:15 xend] INFO (XendDomain:514) Destroying domain:
name=inyoureyes-domain3
[2005-09-16 07:36:15 xend] INFO (XendRoot:141) EVENT> xend.domain.exit
['inyoureyes-domain3', 7, 'crash']
[2005-09-16 07:36:15 xend] INFO (XendRoot:141) EVENT>
xend.domain.destroy ['inyoureyes-domain3', 7]
[2005-09-16 07:36:42 xend] ERROR (XendDomainInfo:128) domain_getinfo(7)
failed, ignoring
Traceback (most recent call last):
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 123, in
dom_get
    domlist = xc.domain_getinfo(dom, 1)
error: (3, 'No such process')
[2005-09-16 07:37:40 xend] ERROR (XendDomainInfo:128) domain_getinfo(7)
failed, ignoring
Traceback (most recent call last):
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 123, in
dom_get
    domlist = xc.domain_getinfo(dom, 1)
error: (3, 'No such process')
[2005-09-16 07:37:40 xend] INFO (XendRoot:141) EVENT> xend.domain.died
['inyoureyes-domain3', None]



This one is easily repeatable at least for my setup.

ERROR: do_evtchn_op: HYPERVISOR_event_channel_op failed: -1
Exception in thread Watcher:
Traceback (most recent call last):
  File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap
    self.run()
  File "/usr/lib/python2.4/threading.py", line 422, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python/xen/xend/xenstore/xswatch.py", line 63, in
watchMain
    watch.fn(*watch.args, **watch.kwargs)
  File "/usr/lib/python/xen/xend/XendDomain.py", line 98, in
onReleaseDomain
    self.refresh(cleanup=True)
  File "/usr/lib/python/xen/xend/XendDomain.py", line 265, in refresh
    self.reap()
  File "/usr/lib/python/xen/xend/XendDomain.py", line 259, in reap
    self.final_domain_destroy(id)
  File "/usr/lib/python/xen/xend/XendDomain.py", line 522, in
final_domain_destroy
    raise XendError(str(ex))
XendError: (3, 'No such process')




Another Caveat
It appears that 

ERROR (process:37) [network-bridge] Cannot find device "xen-br0:1"

Stops happening after a few restarts of xend. 

But this message is consistent for any domains.

(blkif:24) exception looking up device number for sda2: [Errno 2] No
such file or directory: '/dev/sda2'

Seems not to matter as the DomU gets created though.

Also seeing some issue with memory, once the first DomU crashes I am
unable to recover the memory assigned to it. Stopping xend and purging
files in /var/lib/xenstored/store/domain/ no affect. Appears to
cooincide with this trace.

Exception in thread Watcher:
Traceback (most recent call last):
  File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap
    self.run()
  File "/usr/lib/python2.4/threading.py", line 422, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python/xen/xend/xenstore/xswatch.py", line 63, in
watchMain
    watch.fn(*watch.args, **watch.kwargs)
  File "/usr/lib/python/xen/xend/XendDomain.py", line 98, in
onReleaseDomain
    self.refresh(cleanup=True)
  File "/usr/lib/python/xen/xend/XendDomain.py", line 265, in refresh
    self.reap()
  File "/usr/lib/python/xen/xend/XendDomain.py", line 259, in reap
    self.final_domain_destroy(id)
  File "/usr/lib/python/xen/xend/XendDomain.py", line 522, in
final_domain_destroy
    raise XendError(str(ex))
XendError: (3, 'No such process')


Regards,
Ted







_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.