[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Xen-devel] Xen 4.0.4, kernel 3.5.0 HVM crash and kernel BUG
Hi,
Another update:
I wanted to check that if a Linux HVM would boot with working VNC console, so I tried to launch a Debian Squeeze installer on HVM. It refused to start ant told me that vbd hotplug scripts were not working. After that failure even the Windows domU would not anymore start which was previously starting ok.
The hotplug scripts also starts hanging on the processes.
root 9401 0.1 0.1 17700 1640 ? S 15:05 0:00 /bin/bash /etc/xen/scripts/xen-hotplug-cleanup
root 9441 0.1 0.1 17700 1644 ? S 15:05 0:00 /bin/bash /etc/xen/scripts/xen-hotplug-cleanup
root 9481 0.1 0.1 17700 1640 ? S 15:05 0:00 /bin/bash /etc/xen/scripts/xen-hotplug-cleanup
root 9560 0.1 0.1 17700 1640 ? S 15:05 0:00 /bin/bash /etc/xen/scripts/xen-hotplug-cleanup
root 10738 0.1 0.1 17696 1636 ? S 15:05 0:00 /bin/bash /etc/xen/scripts/xen-hotplug-cleanup
root 10747 0.1 0.1 17792 1736 ? S 15:05 0:00 /bin/bash /etc/xen/scripts/block remove
root 11286 0.0 0.0 4080 324 ? S 15:06 0:00 sleep 1
root 11290 0.0 0.0 4080 324 ? S 15:06 0:00 sleep 1
root 11294 0.0 0.0 4080 324 ? S 15:06 0:00 sleep 1
root 11298 0.0 0.0 4080 324 ? S 15:06 0:00 sleep 1
root 11302 0.0 0.0 4080 320 ? S 15:06 0:00 sleep 1
root 11306 0.0 0.0 4080 320 ? S 15:06 0:00 sleep 1
Then I did a xm destroy and I had again the kernel BUG on dmesg. So it seems that the problem is not fixed by using 3.6.0. Udev version is 175-7.
2012/10/1 Valtteri Kiviniemi <kiviniemi.valtteri@xxxxxxxxx>
Hi,
CPU: Intel Core i7-3770 3.4GHz
http://ark.intel.com/products/65719/
MB: Intel DQ77MK (latest bios updated)
http://www.intel.com/content/www/us/en/motherboards/desktop-motherboards/desktop-board-dq77mk.html
Memory: 32GB (4 x 8GB DDR3-1600MHz)
Host is Debian wheezy/testing, Xen 4.0.4 and latest 3.6.0 kernel.
Noticed also some errors in xm dmesg:
root@xen-2:~# xm dmesg
(XEN) Xen version 4.0.4 (root@xxxxxxxxxxxx) (gcc version 4.7.1 (Debian 4.7.1-7) ) Sun Sep 30 20:28:26 EEST 2012
(XEN) Latest ChangeSet: unavailable
(XEN) Bootloader: GNU GRUB 0.97
(XEN) Command line: dom0_mem=1280M cpufreq=xen clocksource=hpet
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: none; EDID transfer time: 0 seconds
(XEN) EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN) Found 4 MBR signatures
(XEN) Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 000000000009d800 (usable)
(XEN) 000000000009d800 - 00000000000a0000 (reserved)
(XEN) 00000000000e0000 - 0000000000100000 (reserved)
(XEN) 0000000000100000 - 0000000020000000 (usable)
(XEN) 0000000020000000 - 0000000020200000 (reserved)
(XEN) 0000000020200000 - 0000000040004000 (usable)
(XEN) 0000000040004000 - 0000000040005000 (reserved)
(XEN) 0000000040005000 - 00000000dbe44000 (usable)
(XEN) 00000000dbe44000 - 00000000dc2d7000 (reserved)
(XEN) 00000000dc2d7000 - 00000000dc2e7000 (ACPI data)
(XEN) 00000000dc2e7000 - 00000000dc40c000 (ACPI NVS)
(XEN) 00000000dc40c000 - 00000000dc6af000 (reserved)
(XEN) 00000000dc6af000 - 00000000dc6b0000 (usable)
(XEN) 00000000dc6b0000 - 00000000dc6f3000 (ACPI NVS)
(XEN) 00000000dc6f3000 - 00000000dd000000 (usable)
(XEN) 00000000dd800000 - 00000000dfa00000 (reserved)
(XEN) 00000000f8000000 - 00000000fc000000 (reserved)
(XEN) 00000000fec00000 - 00000000fec01000 (reserved)
(XEN) 00000000fed00000 - 00000000fed04000 (reserved)
(XEN) 00000000fed1c000 - 00000000fed20000 (reserved)
(XEN) 00000000fee00000 - 00000000fee01000 (reserved)
(XEN) 00000000ff000000 - 0000000100000000 (reserved)
(XEN) 0000000100000000 - 000000081e600000 (usable)
(XEN) ACPI: RSDP 000F0490, 0024 (r2 INTEL)
(XEN) ACPI: XSDT DC2DB080, 007C (r1 INTEL DQ77MK 32 AMI 10013)
(XEN) ACPI: FACP DC2E51F0, 010C (r5 INTEL DQ77MK 32 AMI 10013)
(XEN) ACPI Warning (tbfadt-0232): FADT (revision 5) is longer than ACPI 2.0 version, truncating length 0x10C to 0xF4 [20070126]
(XEN) ACPI: DSDT DC2DB188, A061 (r2 INTEL DQ77MK 32 INTL 20051117)
(XEN) ACPI: FACS DC40A080, 0040
(XEN) ACPI: APIC DC2E5300, 0092 (r3 INTEL DQ77MK 32 AMI 10013)
(XEN) ACPI: FPDT DC2E5398, 0044 (r1 INTEL DQ77MK 32 AMI 10013)
(XEN) ACPI: TCPA DC2E53E0, 0032 (r2 INTEL DQ77MK 32 MSFT 1000013)
(XEN) ACPI: MCFG DC2E5418, 003C (r1 INTEL DQ77MK 32 MSFT 97)
(XEN) ACPI: HPET DC2E5458, 0038 (r1 INTEL DQ77MK 32 AMI. 5)
(XEN) ACPI: SSDT DC2E5490, 036D (r1 INTEL DQ77MK 32 INTL 20091112)
(XEN) ACPI: SSDT DC2E5800, 09AA (r1 INTEL DQ77MK 32 INTL 20051117)
(XEN) ACPI: SSDT DC2E61B0, 0A92 (r1 INTEL DQ77MK 32 INTL 20051117)
(XEN) ACPI: DMAR DC2E6C48, 00B8 (r1 INTEL DQ77MK 32 INTL 1)
(XEN) ACPI: ASF! DC2E6D00, 00A5 (r32 INTEL DQ77MK 32 TFSM F4240)
(XEN) System RAM: 32682MB (33467320kB)
(XEN) Domain heap initialised
(XEN) ACPI: 32/64X FACS address mismatch in FADT - dc40a080/0000000000000000, using 32
(XEN) Processor #0 7:10 APIC version 21
(XEN) Processor #2 7:10 APIC version 21
(XEN) Processor #4 7:10 APIC version 21
(XEN) Processor #6 7:10 APIC version 21
(XEN) Processor #1 7:10 APIC version 21
(XEN) Processor #3 7:10 APIC version 21
(XEN) Processor #5 7:10 APIC version 21
(XEN) Processor #7 7:10 APIC version 21
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) Switched to APIC driver x2apic_cluster.
(XEN) x2APIC mode enabled.
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 3392.369 MHz processor.
(XEN) Initing memory sharing.
(XEN) VMX: Supported advanced features:
(XEN) - APIC MMIO access virtualisation
(XEN) - APIC TPR shadow
(XEN) - Extended Page Tables (EPT)
(XEN) - Virtual-Processor Identifiers (VPID)
(XEN) - Virtual NMI
(XEN) - MSR direct-access bitmap
(XEN) - Unrestricted Guest
(XEN) EPT supports 2MB super page.
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging detected.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) I/O virtualisation enabled
(XEN) - Dom0 mode: Relaxed
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) Total of 8 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using old ACK method
(XEN) TSC is reliable, synchronization unnecessary
(XEN) Platform timer appears to have unexpectedly wrapped 1 times.
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 16 KiB.
(XEN) Brought up 8 CPUs
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1ae7000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 0000000804000000->0000000806000000 (319488 pages to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff81000000->ffffffff81ae7000
(XEN) Init. ramdisk: ffffffff81ae7000->ffffffff81ae7000
(XEN) Phys-Mach map: ffffffff81ae7000->ffffffff81d67000
(XEN) Start info: ffffffff81d67000->ffffffff81d674b4
(XEN) Page tables: ffffffff81d68000->ffffffff81d7b000
(XEN) Boot stack: ffffffff81d7b000->ffffffff81d7c000
(XEN) TOTAL: ffffffff80000000->ffffffff82000000
(XEN) ENTRY ADDRESS: ffffffff815e3210
(XEN) Dom0 has maximum 8 VCPUs
(XEN) [VT-D]iommu.c:718: BIOS did not enable IGD for VT properly. Disabling IGD VT-d engine.
(XEN) Scrubbing Free RAM: done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 172kB init memory.
(XEN) traps.c:2333:d0 Domain attempted WRMSR 000000000000008b from 00000012:00000000 to 00000000:00000000.
- Valtteri
2012/10/1 Pasi Kärkkäinen <pasik@xxxxxx>
On Mon, Oct 01, 2012 at 09:12:50PM +0300, Valtteri Kiviniemi wrote:
> Hi,
>
> Lowering memory or vcpu's does not help, problem is the same. I originally
> installed Xen 4.2.0 and the problem was same on that too. Then I
> downgraded back to 4.0.4 since I thought that this might be a bug on
> 4.2.0. I have been previously running Windows Server 2008 R2 succesfully
> on Xen 4.0.x on different hardware with this same config. Hypervisor is
> 64bit and windows is 64bit.
>
> Any ideas what to try next?
>
What kind of hardware is that?
xen.org automated testing regularly tests Windows VMs, and it works OK there.
-- Pasi
> Ps. qemu-dm.log is the following:
>
> domid: 10
> config qemu network with xen bridge for tap10.0 xenbr0
> Using file /dev/virtuals/ts in read-write mode
> Using file /media/iso/windows_server_2008_r2_sp1.iso in read-only mode
> Watching /local/domain/0/device-model/10/logdirty/cmd
> Watching /local/domain/0/device-model/10/command
> qemu_map_cache_init nr_buckets = 10000 size 4194304
> shared page at pfn feffd
> buffered io page at pfn feffb
> Guest uuid = 52f19e23-2955-c27d-a22c-60c5d8c60d5a
> Time offset set 0
> populating video RAM at ff000000
> mapping video RAM from ff000000
> Register xen platform.
> Done register platform.
> platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw
> state.
> xs_read(/local/domain/0/device-model/10/xen_extended_power_mgmt): read
> error
> medium change watch on `hdc' (index: 1):
> /media/iso/windows_server_2008_r2_sp1.iso
> I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
> Log-dirty: no command yet.
> xs_read(/local/domain/10/log-throttling): read error
> qemu: ignoring not-understood drive `/local/domain/10/log-throttling'
> medium change watch on `/local/domain/10/log-throttling' - unknown device,
> ignored
> cirrus vga map change while on lfb mode
> mapping vram to f0000000 - f0400000
> platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw
> state.
> platform_fixed_ioport: changed ro/rw state of ROM memory area. now is ro
> state.
>
> 2012/10/1 Pasi Kärkkäinen <[1] pasik@xxxxxx>
>
> On Mon, Oct 01, 2012 at 07:23:44PM +0300, Valtteri Kiviniemi wrote:
> > Hi,
> >
> > I have tried other config files, but the problem is the same. At
> the
> > moment I'm using a config file from another server where I have a
> working
> > Windows Server 2008 R2 installation, so I dont think that there is
> > anything wrong with my config:
> >
>
> Did you try with less vcpus, for example 2 ?
> how about with less memory, say 2 GB ?
>
> Did you try with later Xen versions? Is that a 32bit Xen, or 64bit Xen
> hypervisor?
>
> -- Pasi
> > kernel = "/usr/lib/xen/boot/hvmloader"
> > builder = "hvm"
> > shadow_memory = "8"
> > memory = "4096"
> > name = "ts"
> > vcpus = "8"
> > cpus = ["0", "1", "2", "3", "4", "5", "6", "7"]
> > pae = "1"
> > acpi = "1"
> > apic = "1"
> > vfb = [ 'type=vnc, vnclisten=10.100.100.50, vncpasswd=xxx' ]
> > xen_extended_power_mgmt = "0"
> > vif = [ "type=ioemu, mac=00:16:3e:d7:d7:5d, bridge=xenbr0" ]
> > disk = [ "phy:/dev/virtuals/ts,hda,w",
> > "file:/media/iso/windows_server_2008_r2_sp1.iso,hdc:cdrom,r" ]
> > >
> > >
> > >
> > viridian = "1"
> > device_model = "/usr/lib/xen/bin/qemu-dm"
> > boot = "dc"
> > snapshot = "0"
> > vncconsole = "1"
> > sdl = "0"
> > opengl = "0"
> > vnc = "1"
> > nographic = "0"
> > stdvga = "0"
> > tsc_mode = "1"
> > monitor = "0"
> > localtime = "1"
> > usb = "0"
> > keymap = "fi"
> > xen_platform_pci = "1"
> > pci_msitranslate = "1"
> > pci_power_mgmt = "0"
> >
> > 2012/10/1 Pasi Kärkkäinen <[1][2] pasik@xxxxxx>
> >
> > On Mon, Oct 01, 2012 at 06:46:08PM +0300, Valtteri Kiviniemi
> wrote:
> > > Hi,
> > >
> > > Yes, I have viridian=1 on my domU config.
> > >
> >
> > Try with some known good domU configfile.
> >
> > -- Pasi
> > >
> > > On Mon, Oct 01, 2012 at 05:06:53PM +0300, Valtteri
> Kiviniemi
> > wrote:
> > > > Hi,
> > > >
> > > > I'm now using 3.6.0 and can't reproduce that crash
> anymore,
> > so it
> > > seems
> > > > that it was a kernel bug.
> > > >
> > >
> > > OK.
> > > > However I'm still getting black screen on VNC
> > > > when trying to install Windows Server 2008 R2. I can
> see the
> > > "windows is
> > > > loading files" screen but after the installer starts
> the VNC
> > > display goes
> > > > black.
> > > >
> > > > Any ideas?
> > > >
> > >
> > > Do you have viridian=1 specified for the windows vm?
> > >
> > > -- Pasi
> > >
> > > > - Valtteri
> > > >
> > > > 2012/10/1 Pasi Kärkkäinen <[1][2][3][4]pasik@xxxxxx>
> > > >
> > > > On Sun, Sep 30, 2012 at 11:18:03PM +0300, Valtteri
> > Kiviniemi
> > > wrote:
> > > > > Hi,
> > > > >
> > > >
> > > > Hello,
> > > > > I'm trying to get Windows Server 2008 R2
> installation
> > > booting on
> > > > Xen
> > > > > 4.0.4. Using the following config:
> > > > >
> > > >
> > > > <snip>
> > > > >
> > > > > The domU will start booting just fine, but
> after a
> > few
> > > minutes the
> > > > VNC
> > > > > screen goes black. After that when typing "xm
> destroy
> > ts" it
> > > will
> > > > trigger
> > > > > a kernel BUG:
> > > > >
> > > > > BUG: unable to handle kernel NULL pointer
> dereference
> > at
> > > > 0000000000000030
> > > > > IP: [<ffffffff810c50c4>] iput+0x3e/0x195
> > > > > PGD 0
> > > > > Oops: 0000 [#1] SMP
> > > > > CPU 6
> > > > > Pid: 3571, comm: qemu-dm Not tainted
> 3.5.0-dom0 #1
> > > >
> > > > First of all upgrade to latest 3.5.x Linux kernel
> release
> > .. so
> > > at least
> > > > 3.5.4.
> > > >
> > > > -- Pasi
> > > >
> > > > > /DQ77MK
> > > > > RIP: e030:[<ffffffff810c50c4>]
> [<ffffffff810c50c4>]
> > > > iput+0x3e/0x195
> > > > > RSP: e02b:ffff8800389ffbf8 EFLAGS: 00010246
> > > > > RAX: 0000000000000001 RBX: ffff8800377b0720
> RCX:
> > > ffff8800501c0000
> > > > > RDX: ffff8800501c0000 RSI: ffff8800377b0790
> RDI:
> > > ffff8800377b0790
> > > > > RBP: 0000000000000000 R08: ffffffff815cdd00
> R09:
> > > 0000000000000016
> > > > > R10: fefefefefefefeff R11: ffff8800377b0400
> R12:
> > > 00000001000a3e0c
> > > > > R13: 0000000000000000 R14: 00000001000a3e0c
> R15:
> > > ffff8800389ffc28
> > > > > FS: 00007f1af70a8700(0000)
> GS:ffff880050180000(0000)
> > > > > knlGS:0000000000000000
> > > > > CS: e033 DS: 0000 ES: 0000 CR0:
> 000000008005003b
> > > > > CR2: 0000000000000030 CR3: 000000000156d000
> CR4:
> > > 0000000000002660
> > > > > DR0: 0000000000000000 DR1: 0000000000000000
> DR2:
> > > 0000000000000000
> > > > > DR3: 0000000000000000 DR6: 00000000ffff0ff0
> DR7:
> > > 0000000000000400
> > > > > Process qemu-dm (pid: 3571, threadinfo
> > ffff8800389fe000,
> > > task
> > > > > ffff88003a721260)
> > > > > Stack:
> > > > > ffff88003a6d6400 ffff8800377b0000
> 00000001000a3e0c
> > > > ffffffff8133ce8f
> > > > > ffff8800377b0400 ffffffff8134b6cd
> ffff8800389ffc28
> > > > ffff8800389ffc28
> > > > > ffff8800377b00f8 ffff8800377b0680
> ffff880038cdcd60
> > > > ffff8800377b0000
> > > > > Call Trace:
> > > > > [<ffffffff8133ce8f>] ?
> sk_release_kernel+0x23/0x39
> > > > > [<ffffffff8134b6cd>] ?
> netdev_run_todo+0x1e9/0x206
> > > > > [<ffffffff8129798f>] ?
> tun_chr_close+0x4c/0x7b
> > > > > [<ffffffff810b39d3>] ? fput+0xe4/0x1c5
> > > > > [<ffffffff810b202c>] ? filp_close+0x61/0x68
> > > > > [<ffffffff81035e62>] ?
> put_files_struct+0x62/0xb9
> > > > > [<ffffffff81036374>] ? do_exit+0x24a/0x74c
> > > > > [<ffffffff81036906>] ?
> do_group_exit+0x6b/0x9d
> > > > > [<ffffffff8103ea0b>] ?
> > get_signal_to_deliver+0x449/0x46e
> > > > > [<ffffffff81009fa5>] ? do_signal+0x28/0x4c4
> > > > > [<ffffffff81027079>] ?
> > pvclock_clocksource_read+0x48/0xbf
> > > > > [<ffffffff8105b745>] ? ktime_get_ts+0x66/0xa8
> > > > > [<ffffffff810bfb18>] ?
> > poll_select_copy_remaining+0xe0/0xf5
> > > > > [<ffffffff8100a48d>] ?
> do_notify_resume+0x3b/0x74
> > > > > [<ffffffff81411a70>] ? int_signal+0x12/0x17
> > > > > Code: 00 00 00 40 74 02 0f 0b 48 8d 77 70 48
> 8d bf 08
> > 01 00
> > > 00 e8
> > > > 8b 71 10
> > > > > 00 85 c0 0f 84 5d 01 00 00 48 8b 6b 18 f6 83
> 80 00 00
> > 00 08
> > > <4c> 8b
> > > > 65 30
> > > > > 74 11 be 68 05 00 00 48 c7 c7 8e df 4f 81 e8
> bb d0
> > > > > RIP [<ffffffff810c50c4>] iput+0x3e/0x195
> > > > > RSP <ffff8800389ffbf8>
> > > > > CR2: 0000000000000030
> > > > > ---[ end trace 67cc1654658fedcc ]---
> > > > > Fixing recursive fault but reboot is needed!
> > > > >
> > > > > I also tested Xen 4.2.0 and problem is the
> same. So
> > is this
> > > a Xen
> > > > bug or a
> > > > > kernel bug? I am running vanilla
> > > 3.5.0 and
> > > > my
> > > > > hardware is Intel Core i7-3770 CPU and Intel
> DQ77MK
> > > motherboard
> > > > with
> > > > > latest bios.
> > > > >
> > > > > Best regards,
> > > > > Valtteri Kiviniemi
> > > > >
> > > > > References
> > > > >
> > > > > Visible links
> > > > > 1. [3][4][5][6]http://kernel.org/
> > > >
> > > > > _______________________________________________
> > > > > Xen-devel mailing list
> > > > > [4][5][6][7]Xen-devel@xxxxxxxxxxxxx
> > > > > [5][6][7][8]http://lists.xen.org/xen-devel
> > > >
> > > > References
> > > >
> > > > Visible links
> > > > 1. mailto:[7][8][9]pasik@xxxxxx
> > > > 2. [8][9][10]http://kernel.org/
> > > > 3. [9][10][11]http://kernel.org/
> > > > 4. mailto:[10][11][12]Xen-devel@xxxxxxxxxxxxx
> > > > 5. [11][12][13]http://lists.xen.org/xen-devel
> > >
> > > References
> > >
> > > Visible links
> > > 1. mailto:[13][14]pasik@xxxxxx
> > > 2. mailto:[14][15]pasik@xxxxxx
> > > 3. [15][16]http://kernel.org/
> > > 4. [16][17]http://kernel.org/
> > > 5. mailto:[17][18]Xen-devel@xxxxxxxxxxxxx
> > > 6. [18][19]http://lists.xen.org/xen-devel
> > > 7. mailto:[19][20]pasik@xxxxxx
> > > 8. [20][21]http://kernel.org/
> > > 9. [21][22]http://kernel.org/
> > > 10. mailto:[22][23]Xen-devel@xxxxxxxxxxxxx
> > > 11. [23][24]http://lists.xen.org/xen-devel
> >
> > References
> >
> > Visible links
> > 1. mailto:[25]pasik@xxxxxx
> > 2. mailto:[26]pasik@xxxxxx
> > 3. mailto:[27]pasik@xxxxxx
> > 4. [28]http://kernel.org/
> > 5. [29]http://kernel.org/
> > 6. mailto:[30]Xen-devel@xxxxxxxxxxxxx
> > 7. [31]http://lists.xen.org/xen-devel
> > 8. mailto:[32]pasik@xxxxxx
> > 9. [33]http://kernel.org/
> > 10. [34]http://kernel.org/
> > 11. mailto:[35]Xen-devel@xxxxxxxxxxxxx
> > 12. [36]http://lists.xen.org/xen-devel
> > 13. mailto:[37]pasik@xxxxxx
> > 14. mailto:[38]pasik@xxxxxx
> > 15. [39]http://kernel.org/
> > 16. [40]http://kernel.org/
> > 17. mailto:[41]Xen-devel@xxxxxxxxxxxxx
> > 18. [42]http://lists.xen.org/xen-devel
> > 19. mailto:[43]pasik@xxxxxx
> > 20. [44]http://kernel.org/
> > 21. [45]http://kernel.org/
> > 22. mailto:[46]Xen-devel@xxxxxxxxxxxxx
> > 23. [47]http://lists.xen.org/xen-devel
> 4. mailto:pasik@xxxxxx
> 5. http://kernel.org/
> 6. http://kernel.org/
> 7. mailto:Xen-devel@xxxxxxxxxxxxx
> 8. http://lists.xen.org/xen-devel
> 9. mailto:pasik@xxxxxx
> 10. http://kernel.org/
> 11. http://kernel.org/
> 12. mailto:Xen-devel@xxxxxxxxxxxxx
> 13. http://lists.xen.org/xen-devel
> 14. mailto:pasik@xxxxxx
> 15. mailto:pasik@xxxxxx
> 16. http://kernel.org/
> 17. http://kernel.org/
> 18. mailto:Xen-devel@xxxxxxxxxxxxx
> 19. http://lists.xen.org/xen-devel
> 20. mailto:pasik@xxxxxx
> 21. http://kernel.org/
> 22. http://kernel.org/
> 23. mailto:Xen-devel@xxxxxxxxxxxxx
> 24. http://lists.xen.org/xen-devel
> 25. mailto:pasik@xxxxxx
> 26. mailto:pasik@xxxxxx
> 27. mailto:pasik@xxxxxx
> 28. http://kernel.org/
> 29. http://kernel.org/
> 30. mailto:Xen-devel@xxxxxxxxxxxxx
> 31. http://lists.xen.org/xen-devel
> 32. mailto:pasik@xxxxxx
> 33. http://kernel.org/
> 34. http://kernel.org/
> 35. mailto:Xen-devel@xxxxxxxxxxxxx
> 36. http://lists.xen.org/xen-devel
> 37. mailto:pasik@xxxxxx
> 38. mailto:pasik@xxxxxx
> 39. http://kernel.org/
> 40. http://kernel.org/
> 41. mailto:Xen-devel@xxxxxxxxxxxxx
> 42. http://lists.xen.org/xen-devel
> 43. mailto:pasik@xxxxxx
> 44. http://kernel.org/
> 45. http://kernel.org/
> 46. mailto:Xen-devel@xxxxxxxxxxxxx
> 47. http://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel
|
