[Xen-users] Multiple "Domain-0" â slow libvirt

[ÐÐÐÑÑÐÐ ÐÐÐÑÐÐÐÐÐÐÐ <dmitry.b.ogorodnikov@xxxxxxxxx>]

Hi all,

I have a problem with libvirt, it works slow. For example, "virsh
list" takes 6..7 seconds.

I found this in libvirt log:

...
11:38:12.765: debug : virEventDispatchHandles:477 : Dispatch n=8 f=56
w=9 e=1 0x25ce830
11:38:12.765: debug : virEventUpdateHandleImpl:147 : Update handle w=8 e=3
11:38:12.765: debug : virEventInterruptLocked:668 : Interrupting
11:38:12.766: debug : xenStoreDomainIntroduced:1421 : Some domains
were missing, trying again
11:38:12.867: debug : xenStoreDomainIntroduced:1421 : Some domains
were missing, trying again
11:38:12.968: debug : xenStoreDomainIntroduced:1421 : Some domains
were missing, trying again
(about 20 such entries)
11:38:14.681: debug : xenStoreDomainIntroduced:1421 : Some domains
were missing, trying again
11:38:14.782: debug : virEventDispatchHandles:464 : i=9 w=10
11:38:14.782: debug : virEventCleanupTimeouts:495 : Cleanup 3
11:38:14.782: debug : virEventCleanupHandles:536 : Cleanupo 10
...

That's my time spent.

Now look at xenstored config (via script from
http://wiki.xensource.com/xenwiki/XenStore)

/vm/00000000-0000-0000-0000-000000000000/on_xend_stop=ignore
/vm/00000000-0000-0000-0000-000000000000/shadow_memory=0
/vm/00000000-0000-0000-0000-000000000000/uuid=00000000-0000-0000-0000-000000000000
/vm/00000000-0000-0000-0000-000000000000/on_reboot=restart
/vm/00000000-0000-0000-0000-000000000000/image/ostype=linux
/vm/00000000-0000-0000-0000-000000000000/image/kernel=
/vm/00000000-0000-0000-0000-000000000000/image/cmdline=
/vm/00000000-0000-0000-0000-000000000000/image/ramdisk=
/vm/00000000-0000-0000-0000-000000000000/on_poweroff=destroy
/vm/00000000-0000-0000-0000-000000000000/bootloader_args=
/vm/00000000-0000-0000-0000-000000000000/on_xend_start=ignore
/vm/00000000-0000-0000-0000-000000000000/on_crash=restart
/vm/00000000-0000-0000-0000-000000000000/xend/restart_count=0
/vm/00000000-0000-0000-0000-000000000000/vcpus=4
/vm/00000000-0000-0000-0000-000000000000/vcpu_avail=15
/vm/00000000-0000-0000-0000-000000000000/bootloader=
/vm/00000000-0000-0000-0000-000000000000/name=Domain-0
/vm/00000000-0000-0000-0000-000000000000-1/on_xend_stop=ignore
/vm/00000000-0000-0000-0000-000000000000-1/shadow_memory=0
/vm/00000000-0000-0000-0000-000000000000-1/uuid=00000000-0000-0000-0000-000000000000
/vm/00000000-0000-0000-0000-000000000000-1/on_reboot=restart
/vm/00000000-0000-0000-0000-000000000000-1/image/ostype=linux
/vm/00000000-0000-0000-0000-000000000000-1/image/kernel=
/vm/00000000-0000-0000-0000-000000000000-1/image/cmdline=
/vm/00000000-0000-0000-0000-000000000000-1/image/ramdisk=
/vm/00000000-0000-0000-0000-000000000000-1/on_poweroff=destroy
/vm/00000000-0000-0000-0000-000000000000-1/bootloader_args=
/vm/00000000-0000-0000-0000-000000000000-1/on_xend_start=ignore
/vm/00000000-0000-0000-0000-000000000000-1/on_crash=restart
/vm/00000000-0000-0000-0000-000000000000-1/xend/restart_count=0
/vm/00000000-0000-0000-0000-000000000000-1/vcpus=4
/vm/00000000-0000-0000-0000-000000000000-1/vcpu_avail=15
/vm/00000000-0000-0000-0000-000000000000-1/bootloader=
/vm/00000000-0000-0000-0000-000000000000-1/name=Domain-0
/local/domain/0/vm=/vm/00000000-0000-0000-0000-000000000000-1
/local/domain/0/device=
...

I have two (sometimes more) "Domain-0" vms. I think every reboot adds a new one.

I run "xenstore-rm /vm/00000000-0000-0000-0000-000000000000",
"xenstore-rm /vm/00000000-0000-0000-0000-000000000000-1" and so on for
every entry. After reboot I have one "Domain-0" and "virsh list" takes
0,4 second until next reboot.

Part of xend.log (reboot)::

[2012-03-21 12:17:51 1768] INFO (SrvDaemon:220) Xend exited with status 0.
[2012-03-21 12:20:53 1791] INFO (SrvDaemon:332) Xend Daemon started
[2012-03-21 12:20:53 1791] INFO (SrvDaemon:336) Xend changeset: unavailable.
[2012-03-21 12:20:55 1791] DEBUG (XendDomainInfo:149)
XendDomainInfo.recreate({'max_vcpu_id': 3, 'cpu_time': 10554784758L,
'ssidref': 0, 'hvm': 0, 'shutdown_reason': 255, 'dying': 0,
'online_vcpus': 4, 'domid': 0, 'paused': 0, 'crashed': 0, 'running':
1, 'maxmem_kb': 17179869180L, 'shutdown': 0, 'mem_kb': 3400960L,
'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'blocked':
0, 'name': 'Domain-0'})
[2012-03-21 12:20:55 1791] INFO (XendDomainInfo:167) Recreating domain
0, UUID 00000000-0000-0000-0000-000000000000. at /local/domain/0
[2012-03-21 12:20:55 1791] DEBUG (XendDomainInfo:3400) Storing VM
details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid':
'00000000-0000-0000-0000-000000000000', 'on_reboot': 'restart',
'image': '(linux (kernel ) (superpages 0) (tsc_mode 0) (nomigrate
0))', 'on_poweroff': 'destroy', 'bootloader_args': '',
'on_xend_start': 'ignore', 'on_crash': 'restart',
'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15',
'bootloader': '', 'name': 'Domain-0'}
[2012-03-21 12:20:55 1791] DEBUG (XendDomainInfo:1804) Storing domain
details: {'cpu/3/availability': 'online', 'description': '',
'console/limit': '1048576', 'memory/target': '3400960',
'cpu/2/availability': 'online', 'vm':
'/vm/00000000-0000-0000-0000-000000000000-1', 'domid': '0',
'cpu/0/availability': 'online', 'cpu/1/availability': 'online',
'control/platform-feature-multiprocessor-suspend': '1',
'console/type': 'xenconsoled', 'name': 'Domain-0'}
[2012-03-21 12:20:55 1791] DEBUG (XendDomain:464) Adding Domain: 0
[2012-03-21 12:20:55 1791] DEBUG (XendDomain:398) number of vcpus to use is 0
[2012-03-21 12:20:55 1791] DEBUG (XendDomainInfo:1891)
XendDomainInfo.handleShutdownWatch
[2012-03-21 12:20:55 1791] INFO (SrvServer:177) unix
path=/var/lib/xend/xend-socket
...

Why xen creates a new vm "Domain-0" instead of using an existing one?
Is it a bug? How can it be fixed?

My software:
root@sl03500008180:~# dpkg -l | grep xen
ii  libxenstore3.0                      4.0.1-4
Xenstore communications library for Xen
ii  linux-image-2.6.32-5-xen-amd64      2.6.32-41
Linux 2.6.32 for 64-bit PCs, Xen dom0 support
ii  xen-hypervisor-4.0-amd64            4.0.1-4
The Xen Hypervisor on AMD64
ii  xen-linux-system-2.6-xen-amd64      2.6.32+29
Xen system with Linux 2.6 for 64-bit PCs (meta-package)
ii  xen-linux-system-2.6.32-5-xen-amd64 2.6.32-41
Xen system with Linux 2.6.32 on 64-bit PCs (meta-package)
ii  xen-qemu-dm-4.0                     4.0.1-2
Xen Qemu Device Model virtual machine hardware emulator
ii  xen-utils-4.0                       4.0.1-4
XEN administrative tools
ii  xen-utils-common                    4.0.0-1
XEN administrative tools - common files
ii  xenstore-utils                      4.0.1-4
Xenstore utilities for Xen
root@sl03500008180:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 6.0.4 (squeeze)
Release:        6.0.4
Codename:       squeeze
root@sl03500008180:~# uname -a
Linux sl03500008180.opfr.local 2.6.32-5-xen-amd64 #1 SMP Mon Jan 16
20:48:30 UTC 2012 x86_64 GNU/Linux

Regards,
Dmitry

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users