Sorry for the late reply, there is something wrong in network which makes the email not delivered successfully.
The inject method is by using xc_map_foreign_range to map the address of the nested xen and memcpy the trap to the specific location. If there is a problem, it should be the xc_map_foreign_range function, can it map L1 xen's memory to dom0 of L0 xen? It might mapping the memory of nested dom0 instead of the nested xen, as the error says that the max physical address is 0xff000000 when I tried to operate on
memory space of xen using xc_map_foreign_range.
> Since this is an L1 xen crash, the bug is probably in the code that
> tried to inject the trap (vmi_write_8_pa() and callers). It's
> interesting that both the L1 crashes were in unlock routines, but that
> could be a coincidence.
>
> At 11:13 +0800 on 11 Dec (1449832424), quizy_jones@xxxxxxxxxxx wrote:
> > The reproduce method is a little complex. We rely on libvmi to translate virtual addess into physical address and inject traps.
> >
> > Goal: monitor all hypercalls of L1 xen (4.4.1) from dom0 of L0 xen (4.4.6)
> > 1. obtain virtual address of hypercall handler (GVA) from xen-syms-4.4.1 file
> > 2. use vmi_pagetable_lookup of libvmi api to translate GVA into guest physical address (GPA)
> > 3. inject int3 into the first byte of GPA using libvmi api vmi_write_8_pa
> > 4. listen on vmexit events caused by #BP
> > The code of the above procedures is here.
> > By running 'hvm10 nested-xen' and create/destroy any domains in nested-xen, you can reproduce the error. (nested-xen is the domain name of L1 xen)
> >
> > My doubt is whether the second step is right. I.E. How xen manages
> > its own memory translations in nested virtualization? And does the
> > L0 xen have privilege to write the memory of L1 xen? BTW, I'm using
> > (v)EPT for nested xen.
>
> Yes, L0 Xen can write to L1 Xen's memory. Internally, Xen has an EPT
> table for L1 Xen, and mainatins EPT tables that mirror the L1's EPT
> tables (but with the extra translation) for the CPU to use when
> running L2 guests. The code is in xen/arch/x86/mm/hap/nested_*.
>
> The next step is to log everything that goes on around that
> vmi_write_8_pa() call, all the addresses it uses and what data it
> changes, and also modify the L1 hypervisor to print the contents of
> the target in a debug keyhandler, so you can check whether
> vmi_write_8_pa() did what you expected.
>
> Cheers,
>
> Tim.
>
> >
> > Jones
> >
> > From: Konrad Rzeszutek Wilk
> > Date: 2015-12-11 10:17
> > To: quizy_jones@xxxxxxxxxxx
> > CC: xen-devel
> > Subject: Re: [Xen-devel] xen panics when setting int3 traps
> > On Fri, Dec 11, 2015 at 10:01:13AM +0800, quizy_jones@xxxxxxxxxxx wrote:
> > > Aslo reboot when destroy a VM.
> >
> > You need to give more context on how to reproduce this, and you should
> > also CC the maintainers of the code. Please look in MAINTAINERS file.
> >
> > >
> > > The logs from L1 Xen:
> > > (XEN) ----[ Xen-4.4.1 x86_64 debug=n Not tainted ]----
> > > (XEN) CPU: 0
> > > (XEN) RIP: e008:[<ffff82d080127b6b>] _spin_unlock+0x1b/0x30
> > > (XEN) RFLAGS: 0000000000010202 CONTEXT: hypervisor
> > > (XEN) rax: ffff82d0802f8320 rbx: 00007f4b024a9004 rcx: 0000000000000002
> > > (XEN) rdx: ffff82d0802b0000 rsi: 0000000000000080 rdi: 000000007c6ebdb0
> > > (XEN) rbp: ffff82d0802b7e48 rsp: ffff82d0802b7dc0 r8: 0000000000000004
> > > (XEN) r9: 0000000000000002 r10: ffff82d0802284f0 r11: 0000000000000282
> > > (XEN) r12: 0000000000000000 r13: 00007fffd571aff0 r14: ffff8300740a0000
> > > (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 00000000001526f0
> > > (XEN) cr3: 000000003afa0000 cr2: 000000007c6ebdb0
> > > (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008
> > > (XEN) Xen stack trace from rsp=ffff82d0802b7dc0:
> > > (XEN) ffff82d080103285 ffff83007c6eb000 0000000000000000 ffff83007c6eb000
> > > (XEN) 0000000100000005 ffff83007c6eb000 ffff83007c6c1000 ffff82d0802b7ec8
> > > (XEN) ffff82d0802b0000 ffff83007c6eb000 ffff82e000fde500 0000000000000005
> > > (XEN) 0000000000000000 ffff83007ef28000 ffff83007c6c1000 000000000007ef28
> > > (XEN) 00007ff000000003 0000000900000003 0000000001650001 000000000165f670
> > > (XEN) 00007fffd571b140 0000000000000001 00007f4b02299557 0000000000000001
> > > (XEN) 0000000000000000 00007fffd571b0ac 00007f4b01bf2018 000000000165f450
> > > (XEN) 0000000000000001 000000000165f680 00007f4b022a0515 0000000000000000
> > > (XEN) 0000000000000000 0000000000000000 0000000000000001 0000000000000033
> > > (XEN) ffff83007c6c1000 ffff8800442ebec0 ffff88006bdb2490 00007fffd571aff0
> > > (XEN) 00007fffd571aff0 0000000000000000 ffff82d08021aef9 00007fffd571b140
> > > (XEN) 000000000165f670 000000000165b050 000000000165f680 ffff8800442ebec0
> > > (XEN) ffff88006d709400 0000000000000282 00007fff00000001 000000b4d327202c
> > > (XEN) 00007f4b02077040 0000000000000024 ffffffff8100148a 0000000000000000
> > > (XEN) 0000000000000001 00007f4b024a9004 0001010000000000 ffffffff8100148a
> > > (XEN) 000000000000e033 0000000000000282 ffff8800442ebe30 000000000000e02b
> > > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> > > (XEN) 0000000000000000 ffff83007c6c1000 0000000000000000 0000000000000000
> > > (XEN) Xen call trace:
> > > (XEN) [<ffff82d080127b6b>] _spin_unlock+0x1b/0x30
> > > (XEN) [<ffff82d080103285>] do_domctl+0x2c5/0x1180
> > > (XEN) [<ffff82d08021aef9>] syscall_enter+0xa9/0xae
> > > (XEN)
> > > (XEN) Pagetable walk from 000000007c6ebdb0:
> > > (XEN) L4[0x000] = 000000003afaf067 0000000000043139
> > > (XEN) L3[0x001] = 0000000000000000 ffffffffffffffff
> > > (XEN)
> > > (XEN) ****************************************
> > > (XEN) Panic on CPU 0:
> > > (XEN) FATAL PAGE FAULT
> > > (XEN) [error_code=0002]
> > > (XEN) Faulting linear address: 000000007c6ebdb0
> > > (XEN) ****************************************
> > > (XEN)
> > > (XEN) Reboot in five seconds...
> > >
> > > And the logs from L0 Xen (d20 is the L1 Xen):
> > > (d20) HVM Loader
> > > (d20) Detected Xen v4.6.0
> > > (d20) Xenbus rings @0xfeffc000, event channel 1
> > > (d20) System requested SeaBIOS
> > > (d20) CPU speed is 1600 MHz
> > > (d20) Relocating guest memory for lowmem MMIO space disabled
> > > (d20) PCI-ISA link 0 routed to IRQ5
> > > (d20) PCI-ISA link 1 routed to IRQ10
> > > (d20) PCI-ISA link 2 routed to IRQ11
> > > (d20) PCI-ISA link 3 routed to IRQ5
> > > (d20) pci dev 01:3 INTA->IRQ10
> > > (d20) pci dev 02:0 INTA->IRQ11
> > > (d20) pci dev 04:0 INTA->IRQ5
> > > (d20) No RAM in high memory; setting high_mem resource base to 100000000
> > > (d20) pci dev 03:0 bar 10 size 002000000: 0f0000008
> > > (d20) pci dev 02:0 bar 14 size 001000000: 0f2000008
> > > (d20) pci dev 04:0 bar 30 size 000040000: 0f3000000
> > > (d20) pci dev 03:0 bar 30 size 000010000: 0f3040000
> > > (d20) pci dev 03:0 bar 14 size 000001000: 0f3050000
> > > (d20) pci dev 02:0 bar 10 size 000000100: 00000c001
> > > (d20) pci dev 04:0 bar 10 size 000000100: 00000c101
> > > (d20) pci dev 04:0 bar 14 size 000000100: 0f3051000
> > > (d20) pci dev 01:1 bar 20 size 000000010: 00000c201
> > > (d20) Multiprocessor initialisation:
> > > (d20) - CPU0 ... 46-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
> > > (d20) - CPU1 ... 46-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
> > > (d20) Writing SMBIOS tables ...
> > > (d20) Loading SeaBIOS ...
> > > (d20) Creating MP tables ...
> > > (d20) Loading ACPI ...
> > > (d20) vm86 TSS at fc00a180
> > > (d20) BIOS map:
> > > (d20) 10000-100d3: Scratch space
> > > (d20) c0000-fffff: Main BIOS
> > > (d20) E820 table:
> > > (d20) [00]: 00000000:00000000 - 00000000:000a0000: RAM
> > > (d20) HOLE: 00000000:000a0000 - 00000000:000c0000
> > > (d20) [01]: 00000000:000c0000 - 00000000:00100000: RESERVED
> > > (d20) [02]: 00000000:00100000 - 00000000:7f800000: RAM
> > > (d20) HOLE: 00000000:7f800000 - 00000000:fc000000
> > > (d20) [03]: 00000000:fc000000 - 00000001:00000000: RESERVED
> > > (d20) Invoking SeaBIOS ...
> > > (d20) SeaBIOS (version rel-1.8.2-0-g33fbe13-20151206_111754-storage)
> > > (d20)
> > > (d20) Found Xen hypervisor signature at 40000000
> > > (d20) Running on QEMU (i440fx)
> > > (d20) xen: copy e820...
> > > (d20) Relocating init from 0x000de2f0 to 0x7f7ae840 (size 71424)
> > > (d20) CPU Mhz=1601
> > > (d20) Found 7 PCI devices (max PCI bus is 00)
> > > (d20) Allocated Xen hypercall page at 7f7ff000
> > > (d20) Detected Xen v4.6.0
> > > (d20) xen: copy BIOS tables...
> > > (d20) Copying SMBIOS entry point from 0x00010010 to 0x000f6490
> > > (d20) Copying MPTABLE from 0xfc001160/fc001170 to 0x000f6390
> > > (d20) Copying PIR from 0x00010030 to 0x000f6310
> > > (d20) Copying ACPI RSDP from 0x000100b0 to 0x000f62e0
> > > (d20) Using pmtimer, ioport 0xb008
> > > (d20) Scan for VGA option rom
> > > (d20) Running option rom at c000:0003
> > > (d20) pmm call arg1=0
> > > (d20) Turning on vga text mode console
> > > (d20) SeaBIOS (version rel-1.8.2-0-g33fbe13-20151206_111754-storage)
> > > (d20) Machine UUID b99ffc8f-8dda-44c8-b8e7-04331274c410
> > > (d20) All threads complete.
> > > (d20) Found 0 lpt ports
> > > (d20) Found 1 serial ports
> > > (d20) ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
> > > (d20) ATA controller 2 at 170/374/0 (irq 15 dev 9)
> > > (d20) ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (20480 MiBytes)
> > > (d20) Searching bootorder for: /pci@i0cf8/*@1,1/drive@0/disk@0
> > > (d20) PS2 keyboard initialized
> > > (d20) All threads complete.
> > > (d20) Scan for option roms
> > > (d20) Running option rom at c980:0003
> > > (d20) pmm call arg1=1
> > > (d20) pmm call arg1=0
> > > (d20) pmm call arg1=1
> > > (d20) pmm call arg1=0
> > > (d20) Searching bootorder for: /pci@i0cf8/*@4
> > > (d20)
> > > (d20) Press F12 for boot menu.
> > > (d20)
> > > (d20) Searching bootorder for: HALT
> > > (d20) drive 0x000f6290: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=41943040
> > > (d20) Space available for UMB: ca800-ee800, f5cb0-f6290
> > > (d20) Returned 258048 bytes of ZoneHigh
> > > (d20) e820 map has 6 items:
> > > (d20) 0: 0000000000000000 - 000000000009fc00 = 1 RAM
> > > (d20) 1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
> > > (d20) 2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
> > > (d20) 3: 0000000000100000 - 000000007f7ff000 = 1 RAM
> > > (d20) 4: 000000007f7ff000 - 000000007f800000 = 2 RESERVED
> > > (d20) 5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
> > > (d20) enter handle_19:
> > > (d20) NULL
> > > (d20) Booting from Hard Disk...
> > > (d20) Booting from 0000:7c00
> > >
> > >
> > > jones
> > >
> > > From: quizy_jones@xxxxxxxxxxx
> > > Date: 2015-12-11 09:42
> > > To: xen-devel
> > > Subject: [Xen-devel] xen panics when setting int3 traps
> > > I'd like to inject int3 traps to hypercall handlers to be able to capture hypercalls. However, the dom0/xen would reboot whenever I create a new VM. Followup is the console output when this happens.
> > >
> > > (XEN) Xen version 4.4.1 (Ubuntu 4.4.1-0ubuntu0.14.04.6) (stefan.bader@xxxxxxxxxxxxx) (gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2) debug=n Wed May 20 12:19:20 UTC 2015
> > > (XEN) Bootloader: GRUB 2.02~beta2-9ubuntu1.2
> > > (XEN) Command line: placeholder console=com1 com1=115200 dom0_max_vcpus=1 dom0_vcpus_pin
> > > (XEN) Video information:
> > > (XEN) VGA is text mode 80x25, font 8x16
> > > (XEN) Disc information:
> > > (XEN) Found 1 MBR signatures
> > > (XEN) Found 1 EDD information structures
> > > (XEN) Xen-e820 RAM map:
> > > (XEN) 0000000000000000 - 000000000009fc00 (usable)
> > > (XEN) 000000000009fc00 - 00000000000a0000 (reserved)
> > > (XEN) 00000000000f0000 - 0000000000100000 (reserved)
> > > (XEN) 0000000000100000 - 000000007f7ff000 (usable)
> > > (XEN) 000000007f7ff000 - 000000007f800000 (reserved)
> > > (XEN) 00000000fc000000 - 0000000100000000 (reserved)
> > > (XEN) System RAM: 2039MB (2088568kB)
> > > (XEN) ACPI: RSDP 000F62E0, 0024 (r2 Xen)
> > > (XEN) ACPI: XSDT FC00A090, 0054 (r1 Xen HVM 0 HVML 0)
> > > (XEN) ACPI: FACP FC0099C0, 00F4 (r4 Xen HVM 0 HVML 0)
> > > (XEN) ACPI: DSDT FC0012A0, 8691 (r2 Xen HVM 0 INTL 20140214)
> > > (XEN) ACPI: FACS FC001260, 0040
> > > (XEN) ACPI: APIC FC009AC0, 0460 (r2 Xen HVM 0 HVML 0)
> > > (XEN) ACPI: HPET FC009FA0, 0038 (r1 Xen HVM 0 HVML 0)
> > > (XEN) ACPI: WAET FC009FE0, 0028 (r1 Xen HVM 0 HVML 0)
> > > (XEN) ACPI: SSDT FC00A010, 0031 (r2 Xen HVM 0 INTL 20140214)
> > > (XEN) ACPI: SSDT FC00A050, 0031 (r2 Xen HVM 0 INTL 20140214)
> > > (XEN) Domain heap initialised
> > > (XEN) Processor #0 7:15 APIC version 20
> > > (XEN) Processor #2 7:15 APIC version 20
> > > (XEN) IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-47
> > > (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
> > > (XEN) Not enabling x2APIC: depends on iommu_supports_eim.
> > > (XEN) Using scheduler: SMP Credit Scheduler (credit)
> > > (XEN) Detected 1600.055 MHz processor.
> > > (XEN) Initing memory sharing.
> > > (XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7
> > > (XEN) I/O virtualisation disabled
> > > (XEN) ENABLING IO-APIC IRQs
> > > (XEN) -> Using new ACK method
> > > (XEN) Platform timer is 62.500MHz HPET
> > > (XEN) Allocated console ring of 16 KiB.
> > > (XEN) VMX: Supported advanced features:
> > > (XEN) - APIC MMIO access virtualisation
> > > (XEN) - APIC TPR shadow
> > > (XEN) - Extended Page Tables (EPT)
> > > (XEN) - Virtual-Processor Identifiers (VPID)
> > > (XEN) - MSR direct-access bitmap
> > > (XEN) - Unrestricted Guest
> > > (XEN) HVM: ASIDs enabled.
> > > (XEN) HVM: VMX enabled
> > > (XEN) HVM: Hardware Assisted Paging (HAP) detected
> > > (XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
> > > (XEN) Brought up 2 CPUs
> > > (XEN) xenoprof: Initialization failed. Intel processor family 6 model 63is not supported
> > > (XEN) *** LOADING DOMAIN 0 ***
> > > (XEN) Xen kernel: 64-bit, lsb, compat32
> > > (XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x23ff000
> > > (XEN) PHYSICAL MEMORY ARRANGEMENT:
> > > (XEN) Dom0 alloc.: 0000000074000000->0000000078000000 (462231 pages to be allocated)
> > > (XEN) Init. ramdisk: 000000007def5000->000000007f1ff662
> > > (XEN) VIRTUAL MEMORY ARRANGEMENT:
> > > (XEN) Loaded kernel: ffffffff81000000->ffffffff823ff000
> > > (XEN) Init. ramdisk: ffffffff823ff000->ffffffff83709662
> > > (XEN) Phys-Mach map: ffffffff8370a000->ffffffff83aba510
> > > (XEN) Start info: ffffffff83abb000->ffffffff83abb4b4
> > > (XEN) Page tables: ffffffff83abc000->ffffffff83add000
> > > (XEN) Boot stack: ffffffff83add000->ffffffff83ade000
> > > (XEN) TOTAL: ffffffff80000000->ffffffff83c00000
> > > (XEN) ENTRY ADDRESS: ffffffff81d341f0
> > > (XEN) Dom0 has maximum 1 VCPUs
> > > (XEN) Scrubbing Free RAM: .done.
> > > (XEN) Initial low memory virq threshold set at 0x4000 pages.
> > > (XEN) Std. Loglevel: Errors and warnings
> > > (XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
> > > (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
> > > (XEN) Freed 272kB init memory.
> > > mapping kernel into physical memory
> > > about to get started...
> > > --------------------------- logs when xen panics
> > > (XEN) ----[ Xen-4.4.1 x86_64 debug=n Not tainted ]----
> > > (XEN) CPU: 0
> > > (XEN) RIP: e008:[<ffff82d080128132>] _write_unlock+0x22/0x40
> > > (XEN) RFLAGS: 0000000000010246 CONTEXT: hypervisor
> > > (XEN) rax: 0000000080000000 rbx: ffff82e000233440 rcx: 0000000000000000
> > > (XEN) rdx: 0000000000000000 rsi: 00000000000119a2 rdi: ffff83007c6ca130
> > > (XEN) rbp: ffff83007c6eb000 rsp: ffff82d0802b7d40 r8: 0000000000233440
> > > (XEN) r9: 0000000000000000 r10: ffff82d0802284f0 r11: 0000000000000246
> > > (XEN) r12: 0000000000000001 r13: 00000000000119a2 r14: 00000000000119a2
> > > (XEN) r15: ffff82d0802b7d64 cr0: 0000000080050033 cr4: 00000000001526f0
> > > (XEN) cr3: 0000000075c0e000 cr2: 00007f54fd0cfe60
> > > (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008
> > > (XEN) Xen stack trace from rsp=ffff82d0802b7d40:
> > > (XEN) ffff82d080115b98 ffff82e000000001 ffff83007c6eb000 0000000000000001
> > > (XEN) 0000000000000000 ffff82e000eb81c0 0000000000000001 0000000000000000
> > > (XEN) ffff88006d62dd70 ffff82d0802b0000 ffff83007c6eb000 0000000000000000
> > > (XEN) ffff82d080116138 ffff83007c6eb000 ffff82e0002219c0 00000000000110ce
> > > (XEN) ffff83007c6c1000 ffff83007c6eb000 0000000000000000 0000000000000200
> > > (XEN) ffffffff81fbf040 ffffffff00000001 ffff82d0802b0000 ffff82d0802b7e70
> > > (XEN) ffff82d0802b7e80 ffff83007c6eb000 ffff83007c6c1000 ffff82d0802b0000
> > > (XEN) ffff82d000000000 ffff83007c6eb000 ffff880000000000 ffff83007c6c1000
> > > (XEN) 00007ff000000002 ffff83007c6eb000 0000000000000000 ffff83007c6c1000
> > > (XEN) ffff82d08017c29c 00000021a5fe6264 00000000000119a2 ffff82d0802f92c8
> > > (XEN) ffff82d0802b0000 ffff82d0802f8500 ffffffff81fbf040 0000000000000200
> > > (XEN) 0000000000000000 0000000000007ff0 0000000000000000 ffff82d0802cf700
> > > (XEN) ffff82d0802b0000 ffffffffffffffff ffff83007c6c1000 ffff88006d62ddb8
> > > (XEN) ffff83007c6c1000 ffff88006d62ddb8 0000000000000200 000000000006ba25
> > > (XEN) ffffea0000000000 ffffea0001ae8940 ffff82d08021aef9 ffffea0001ae4c80
> > > (XEN) ffffea0000000000 000000000006b932 0000000000000200 ffff88006d62ddb8
> > > (XEN) 0000000000000200 0000000000000246 0000000000007ff0 0000000000010bea
> > > (XEN) 0000000000000000 000000000000000c ffffffff8100118a 0000000000000000
> > > (XEN) ffff88006d62dd70 0000000000000001 0001010000000000 ffffffff8100118a
> > > (XEN) 000000000000e033 0000000000000246 ffff88006d62dd50 000000000000e02b
> > > (XEN) Xen call trace:
> > > (XEN) [<ffff82d080128132>] _write_unlock+0x22/0x40
> > > (XEN) [<ffff82d080115b98>] guest_remove_page+0xf8/0x2e0
> > > (XEN) [<ffff82d080116138>] do_memory_op+0x3b8/0x2750
> > > (XEN) [<ffff82d08017c29c>] __do_update_va_mapping+0x1bc/0x6c0
> > > (XEN) [<ffff82d08021aef9>] syscall_enter+0xa9/0xae
> > > (XEN)
> > > (XEN)
> > > (XEN) ****************************************
> > > (XEN) Panic on CPU 0:
> > > (XEN) FATAL TRAP: vector = 6 (invalid opcode)
> > > (XEN) ****************************************
> > > (XEN)
> > > (XEN) Reboot in five seconds...
> > >
> > > BTW, I'm using a nested xen 4.4.1 on xen 4.6.0.
> > >
> > >
> > > Jones
> >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@xxxxxxxxxxxxx
> > > http://lists.xen.org/xen-devel
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@xxxxxxxxxxxxx
> > http://lists.xen.org/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel
