Re: [Xen-users] Re: XEN 3.1: critical bug: vif init failure after creating 15-17 VMs (XENBUS: Timeout connecting to device: device/vif)

[Keir Fraser <keir@xxxxxxxxxxxxx>]

Hi Thomas,

This problem is entirely different. The problem is visible earlier in your
console output: the Xen block-device driver is unable to acquire the
device-number space for SCSI devices (sda, sdb, etc). Hence it is failing to
initialise the vbd connections to the backend and is ending up in state 6
(which is XenbusStateClosed).

The solutions you have are:
 1. Do not build the generic SCSI subsystem into your dom0 kernels. It is
this subsystem which (quite reasonably) is allocating the sd* number space
to the exclusion of the Xen block-device driver.
 2. Call your devices hd* instead of sd* (i.e., hijack the IDE device
numbers instead of the SCSI ones), or even use the xvd* number space, which
is exclusively reserved for Xen VBDs.

 Hope this helps,
 Keir

On 24/7/07 12:06, "Thomas Ronner" <thomas@xxxxxxxx> wrote:

> Hi Keir,
> 
> Keir Fraser wrote:
>> Now fixed in the staging tree. The patch (for your dom0 kernel) is also
>> attached to this email.
> 
> I have a similar problem with vbds instead of vifs:
> 
> (domU:)
> XENBUS: Timeout connecting to device: device/vbd/2049 (state 6)
> XENBUS: Timeout connecting to device: device/vbd/2052 (state 6)
> XENBUS: Timeout connecting to device: device/vbd/2050 (state 6)
> XENBUS: Timeout connecting to device: device/vbd/2051 (state 6)
> 
> 
> Does your patch also fix this (in theory)? This is a production machine
> so I'm somewhat reluctant to try things before knowing what they do.
> I'll attach the full domU output below. This is using a custom kernel
> without modules (I hate having to deploy modules in all domUs) and
> kernel level IP auto config (I like having this info in the xen config
> file).
> 
> There are other domUs on this machine with similar configs having no
> problem at all.
> 
> 
> 
>>  -- Keir
> 
> Regards, Thomas
> 
> 
> 
> ---8<--[ domU output ]------------------------------------------
> [root@diana ~]# xm create vechtstreek_test -c
> Using config file "/etc/xen/vechtstreek_test".
> Started domain vechtstreek_test
>                                 Linux version 2.6.18-tr01
> (root@xxxxxxxxxxxxxxxxxx) (gcc version 4.1.1 20070105 (Red Hat
> 4.1.1-52)) #2 SMP Fri Jul 20 12:14:40 CEST 2007
> BIOS-provided physical RAM map:
>   Xen: 0000000000000000 - 0000000010800000 (usable)
> 0MB HIGHMEM available.
> 264MB LOWMEM available.
> NX (Execute Disable) protection: active
> Allocating PCI resources starting at 20000000 (gap: 10800000:ef800000)
> Detected 3200.282 MHz processor.
> Built 1 zonelists.  Total pages: 67584
> Kernel command line: root=/dev/sda1 ro
> ip=131.211.84.207:1.2.3.4:131.211.84.193:255.255.255.192:vechtstreek_test:eth0
> :off
> Enabling fast FPU save and restore... done.
> Enabling unmasked SIMD FPU exception support... done.
> Initializing CPU#0
> PID hash table entries: 2048 (order: 11, 8192 bytes)
> Xen reported: 3200.112 MHz processor.
> Console: colour dummy device 80x25
> Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
> Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
> Software IO TLB disabled
> vmalloc area: d1000000-f51fe000, maxmem 2d7fe000
> Memory: 251648k/270336k available (3953k kernel code, 10220k reserved,
> 1648k data, 216k init, 0k highmem)
> Checking if this processor honours the WP bit even in supervisor mode... Ok.
> Calibrating delay using timer specific routine.. 6403.14 BogoMIPS
> (lpj=32015708)
> Security Framework v1.0.0 initialized
> Capability LSM initialized
> Mount-cache hash table entries: 512
> CPU: Trace cache: 12K uops, L1 D cache: 16K
> CPU: L2 cache: 2048K
> Checking 'hlt' instruction... OK.
> SMP alternatives: switching to UP code
> Freeing SMP alternatives: 20k freed
> Brought up 1 CPUs
> migration_cost=0
> checking if image is initramfs... it is
> Freeing initrd memory: 588k freed
> NET: Registered protocol family 16
> Brought up 1 CPUs
> xen_mem: Initialising balloon driver.
> SCSI subsystem initialized
> NET: Registered protocol family 2
> IP route cache hash table entries: 4096 (order: 2, 16384 bytes)
> TCP established hash table entries: 16384 (order: 5, 131072 bytes)
> TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
> TCP: Hash tables configured (established 16384 bind 8192)
> TCP reno registered
> audit: initializing netlink socket (disabled)
> audit(1185274517.008:1): initialized
> VFS: Disk quotas dquot_6.5.1
> Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
> Installing knfsd (copyright (C) 1996 okir@xxxxxxxxxxxx).
> NTFS driver 2.1.27 [Flags: R/O].
> fuse init (API version 7.7)
> OCFS2 1.3.3
> OCFS2 Node Manager 1.3.3
> OCFS2 DLM 1.3.3
> OCFS2 DLMFS 1.3.3
> OCFS2 User DLM kernel interface loaded
> seclvl: seclvl_init: seclvl: Failure registering with the kernel.
> seclvl: seclvl_init: seclvl: Failure registering with primary security
> module.
> seclvl: Error during initialization: rc = [-22]
> Initializing Cryptographic API
> io scheduler noop registered
> io scheduler anticipatory registered
> io scheduler deadline registered
> io scheduler cfq registered (default)
> rtc: IRQ 8 is not free.
> RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize
> loop: loaded (max 8 devices)
> nbd: registered device at major 43
> tun: Universal TUN/TAP device driver, 1.6
> tun: (C) 1999-2004 Max Krasnyansky <maxk@xxxxxxxxxxxx>
> Xen virtual console successfully installed as tty1
> Event-channel device installed.
> netfront: Initialising virtual ethernet driver.
> Loading iSCSI transport class v1.1-646.<5>iscsi: registered transport (tcp)
> register_blkdev: cannot get major 8 for sd
> vbd vbd-2049: 19 xlvbd_add at /local/domain/0/backend/vbd/17/2049
> i8042.c: No controller found.
> mice: PS/2 mouse device common for all mice
> register_blkdev: cannot get major 8 for sd
> vbd vbd-2049: 19 xlvbd_add at /local/domain/0/backend/vbd/17/2049
> device-mapper: ioctl: 4.7.0-ioctl (2006-06-24) initialised:
> dm-devel@xxxxxxxxxx
> device-mapper: multipath: version 1.0.4 loaded
> device-mapper: multipath round-robin: version 1.0.0 loaded
> register_blkdev: cannot get major 8 for sd
> vbd vbd-2052: 19 xlvbd_add at /local/domain/0/backend/vbd/17/2052
> dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-2)
> netem: version 1.2
> u32 classifier
>      Performance counters on
>      OLD policer on
> Netfilter messages via NETLINK v0.30.
> IPv4 over IPv4 tunneling driver
> register_blkdev: cannot get major 8 for sd
> vbd vbd-2052: 19 xlvbd_add at /local/domain/0/backend/vbd/17/2052
> GRE over IPv4 tunneling driver
> ip_conntrack version 2.4 (2112 buckets, 16896 max) - 228 bytes per conntrack
> register_blkdev: cannot get major 8 for sd
> vbd vbd-2050: 19 xlvbd_add at /local/domain/0/backend/vbd/17/2050
> register_blkdev: cannot get major 8 for sd
> vbd vbd-2050: 19 xlvbd_add at /local/domain/0/backend/vbd/17/2050
> register_blkdev: cannot get major 8 for sd
> vbd vbd-2051: 19 xlvbd_add at /local/domain/0/backend/vbd/17/2051
> register_blkdev: cannot get major 8 for sd
> vbd vbd-2051: 19 xlvbd_add at /local/domain/0/backend/vbd/17/2051
> netfront: device eth0 has copying receive path.
> ctnetlink v0.90: registering with nfnetlink.
> ip_conntrack_pptp version 3.1 loaded
> ip_nat_pptp version 3.0 loaded
> ip_tables: (C) 2000-2006 Netfilter Core Team
> ClusterIP Version 0.8 loaded successfully
> arp_tables: (C) 2002 David S. Miller
> IPVS: Registered protocols (TCP, UDP, AH, ESP)
> IPVS: Connection hash table configured (size=4096, memory=32Kbytes)
> IPVS: ipvs loaded.
> IPVS: [rr] scheduler registered.
> IPVS: [wrr] scheduler registered.
> IPVS: [lc] scheduler registered.
> IPVS: [wlc] scheduler registered.
> IPVS: [lblc] scheduler registered.
> IPVS: [lblcr] scheduler registered.
> IPVS: [dh] scheduler registered.
> IPVS: [sh] scheduler registered.
> IPVS: [sed] scheduler registered.
> IPVS: [nq] scheduler registered.
> IPVS: ftp: loaded support on port[0] = 21
> TCP bic registered
> TCP cubic registered
> TCP westwood registered
> TCP highspeed registered
> TCP hybla registered
> TCP htcp registered
> TCP vegas registered
> TCP veno registered
> TCP scalable registered
> TCP lp registered
> Initializing IPsec netlink socket
> NET: Registered protocol family 1
> NET: Registered protocol family 10
> lo: Disabled Privacy Extensions
> IPv6 over IPv4 tunneling driver
> ip6_tables: (C) 2000-2006 Netfilter Core Team
> NET: Registered protocol family 17
> NET: Registered protocol family 15
> Bridge firewalling registered
> Ebtables v2.0 registered
> ebt_ulog: not logging via ulog since somebody else already registered
> for PF_BRIDGE
> 802.1Q VLAN Support v1.8 Ben Greear <greearb@xxxxxxxxxxxxxxx>
> All bugs added by David S. Miller <davem@xxxxxxxxxx>
> ieee80211: 802.11 data/management/control stack, git-1.1.13
> ieee80211: Copyright (C) 2004-2005 Intel Corporation
> <jketreno@xxxxxxxxxxxxxxx>
> Using IPI No-Shortcut mode
> XENBUS: Timeout connecting to device: device/vbd/2049 (state 6)
> XENBUS: Timeout connecting to device: device/vbd/2052 (state 6)
> XENBUS: Timeout connecting to device: device/vbd/2050 (state 6)
> XENBUS: Timeout connecting to device: device/vbd/2051 (state 6)
> XENBUS: Device with no driver: device/console/0
> IP-Config: Complete:
>        device=eth0, addr=131.211.84.207, mask=255.255.255.192,
> gw=131.211.84.193,
>       host=vechtstreek_test, domain=, nis-domain=(none),
>       bootserver=1.2.3.4, rootserver=1.2.3.4, rootpath=
> Freeing unused kernel memory: 216k freed
> Red Hat nash version 4.1.18 starting
> Mounted /proc filesystem
> Mounting sysfs
> Creating /dev
> Starting udev
> Creating root device
> Mounting root filesystem
> mount: error 6 mounting ext3
> mount: error 2 mounting none
> Switching to new root
> switchroot: mount failed: 22
> umount /initrd/dev failed: 2
> Kernel panic - not syncing: Attempted to kill init!
> --------------------------------------------------------
> 
> ---8<--[ /etc/xen/vechtstreek_test ]--------------------
> kernel = "/boot/vmlinux-stripped"
> ramdisk = "/boot/initrd-xenU-tr01"
> memory = 256
> name = "vechtstreek_test"
> vif = [ 'mac=00:00:6C:00:00:0D' ]
> disk = [ 'phy:sata/vechtstreek_root,sda1,w',
>           'phy:sata/vechtstreek_swap,sda4,w',
>           'phy:sata/vechtstreek_var,sda2,w',
>           'phy:sata/vechtstreek_home,sda3,w' ]
> ip="131.211.84.207"
> netmask="255.255.255.192"
> gateway="131.211.84.193"
> hostname="vechtstreek_test"
> root = "/dev/sda1 ro"


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users