[Top] [All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] GPU passthrough issue when VM is configured with 4G memory

Le 13.03.2013 14:23, Hanweidong a ÃcritÂ:

-----Original Message-----
From: dunlapg@xxxxxxxxx [mailto:dunlapg@xxxxxxxxx] On Behalf Of George 
Dunlap
Sent: 2013å3æ12æ 18:40
To: Hanweidong
Cc: Stefano Stabellini; Yanqiangjun; Luonengjun; Wangzhenguo;
Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-devel@xxxxxxxxxxxxx
Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
with 4G memory

On Tue, Mar 12, 2013 at 5:45 AM, Hanweidong <hanweidong@xxxxxxxxxx>
wrote:
> I don't think it's a pass-through issue. We also encountered similar 
> issue when using ivshmem device with 512M mmio. The problem is
hvmloader
> adjusted pci_mem_start according to actual mmio size of VM, but QEMU 
does
> not know the actual mmio size of VM, it just uses the
HVM_BELOW_4G_RAM_END
> and HVM_BELOW_4G_MMIO_LENGTH in xen_ram_init. This results in memory 
layout
> mismatch between hvmloader and QEMU. I think it needs to make QEMU
know
> actual mmio size, and then it can make the same memory layout as
hvmloader.

So I'm not an expert in this codepath, but it was my understanding
that resizing the MMIO hole is actually the job of the guest BIOS: it is supposed to query the PCI devices to find out how much memory they 
need, and then relocate memory as appropriate.  XenServer never had
any problem with the MMIO hole not being big enough when passing cards 
to the guest; the only problem we've ever had (as far as I know) is
the issue I mentioned before, where there was a bug in some pci bridge cards that breaks VT-d if the guest physical address overlaps the host 
physical address range of another device.

Can you maybe boot Linux in an HVM domain and do an lspci or look at
Linux's e820 map, and check to see whether the BIOS has in fact
relocated the guest memory, or whether the MMIO hole does indeed start 
only at HVM_BELOW_4G_RAM_END?
We tried Linux guest, but it stopped at grub screen, cannot boot up. Posted 
"xl dmesg" output as below:

(XEN) HVM13: HVM Loader
(XEN) HVM13: Detected Xen v4.3-unstable
(XEN) HVM13: Xenbus rings @0xfeffc000, event channel 3
(XEN) HVM13: System requested SeaBIOS
(XEN) HVM13: CPU speed is 2500 MHz
(XEN) irq.c:270: Dom13 PCI link 0 changed 0 -> 5
(XEN) HVM13: PCI-ISA link 0 routed to IRQ5
(XEN) irq.c:270: Dom13 PCI link 1 changed 0 -> 10
(XEN) HVM13: PCI-ISA link 1 routed to IRQ10
(XEN) irq.c:270: Dom13 PCI link 2 changed 0 -> 11
(XEN) HVM13: PCI-ISA link 2 routed to IRQ11
(XEN) irq.c:270: Dom13 PCI link 3 changed 0 -> 5
(XEN) HVM13: PCI-ISA link 3 routed to IRQ5
(XEN) HVM13: pci dev 01:2 INTD->IRQ5
(XEN) HVM13: pci dev 01:3 INTA->IRQ10
(XEN) HVM13: pci dev 03:0 INTA->IRQ5
(XEN) HVM13: pci dev 04:0 INTA->IRQ5
(XEN) HVM13: pci dev 05:0 INTA->IRQ10
(XEN) HVM13: pci dev 05:0 bar 14 size lx: 08000000
(XEN) memory_map:add: dom13 gfn=e0000 mfn=e8000 nr=8000
(XEN) memory_map:add: dom13 gfn=e8000 mfn=f0000 nr=4000
(XEN) HVM13: pci dev 05:0 bar 1c size lx: 04000000
(XEN) HVM13: pci dev 02:0 bar 10 size lx: 02000000
(XEN) memory_map:add: dom13 gfn=ee000 mfn=bc000 nr=2000
(XEN) HVM13: pci dev 05:0 bar 10 size lx: 02000000
(XEN) HVM13: pci dev 03:0 bar 14 size lx: 01000000
(XEN) HVM13: pci dev 05:0 bar 30 size lx: 00080000
(XEN) HVM13: pci dev 02:0 bar 30 size lx: 00010000
(XEN) HVM13: pci dev 04:0 bar 30 size lx: 00010000
(XEN) HVM13: pci dev 02:0 bar 14 size lx: 00001000
(XEN) HVM13: pci dev 03:0 bar 10 size lx: 00000100
(XEN) HVM13: pci dev 04:0 bar 10 size lx: 00000100
(XEN) HVM13: pci dev 04:0 bar 14 size lx: 00000100
(XEN) HVM13: pci dev 05:0 bar 24 size lx: 00000080
(XEN) ioport_map:add: dom13 gport=c200 mport=2000 nr=80
(XEN) HVM13: pci dev 01:2 bar 20 size lx: 00000020
(XEN) HVM13: pci dev 01:1 bar 20 size lx: 00000010
(XEN) HVM13: Multiprocessor initialisation:
(XEN) HVM13:  - CPU0 ... 46-bit phys ... fixed MTRRs ... var MTRRs
[3/8] ... done.
(XEN) HVM13: Testing HVM environment:
(XEN) HVM13:  - REP INSB across page boundaries ... passed
(XEN) HVM13:  - GS base MSRs and SWAPGS ... passed
(XEN) HVM13: Passed 2 of 2 tests
(XEN) HVM13: Writing SMBIOS tables ...
(XEN) HVM13: Loading SeaBIOS ...
(XEN) HVM13: Creating MP tables ...
(XEN) HVM13: Loading ACPI ...
(XEN) HVM13: vm86 TSS at fc00a000
(XEN) HVM13: BIOS map:
(XEN) HVM13:  10000-100d3: Scratch space
(XEN) HVM13:  e0000-fffff: Main BIOS
(XEN) HVM13: E820 table:
(XEN) HVM13:  [00]: 00000000:00000000 - 00000000:000a0000: RAM
(XEN) HVM13:  HOLE: 00000000:000a0000 - 00000000:000e0000
(XEN) HVM13:  [01]: 00000000:000e0000 - 00000000:00100000: RESERVED
(XEN) HVM13:  [02]: 00000000:00100000 - 00000000:e0000000: RAM
(XEN) HVM13:  HOLE: 00000000:e0000000 - 00000000:fc000000
(XEN) HVM13:  [03]: 00000000:fc000000 - 00000001:00000000: RESERVED
(XEN) HVM13:  [04]: 00000001:00000000 - 00000001:1f800000: RAM
(XEN) HVM13: Invoking SeaBIOS ...
(XEN) HVM13: SeaBIOS (version rel-1.7.1-0-g51755c3-20130227_164001-linux-DPMZPO) 
(XEN) HVM13:
(XEN) HVM13: Found Xen hypervisor signature at 40000000
(XEN) HVM13: xen: copy e820...
(XEN) HVM13: Ram Size=0xe0000000 (0x000000001f800000 high)
(XEN) HVM13: Relocating low data from 0x000e2490 to 0x000ef790 (size 2156) (XEN) HVM13: Relocating init from 0x000e2cfc to 0xdffe20f0 (size 56804) 
(XEN) HVM13: CPU Mhz=2501
(XEN) HVM13: Found 9 PCI devices (max PCI bus is 00)
(XEN) HVM13: Allocated Xen hypercall page at dffff000
(XEN) HVM13: Detected Xen v4.3-unstable
(XEN) HVM13: Found 1 cpu(s) max supported 1 cpu(s)
(XEN) HVM13: xen: copy BIOS tables...
(XEN) HVM13: Copying SMBIOS entry point from 0x00010010 to 0x000fdb10
(XEN) HVM13: Copying MPTABLE from 0xfc001140/fc001150 to 0x000fda30
(XEN) HVM13: Copying PIR from 0x00010030 to 0x000fd9b0
(XEN) HVM13: Copying ACPI RSDP from 0x000100b0 to 0x000fd980
(XEN) HVM13: Scan for VGA option rom
(XEN) HVM13: Running option rom at c000:0003
(XEN) stdvga.c:147:d13 entering stdvga and caching modes
(XEN) HVM13: Turning on vga text mode console
(XEN) HVM13: SeaBIOS (version rel-1.7.1-0-g51755c3-20130227_164001-linux-DPMZPO) 
(XEN) HVM13:
(XEN) HVM13: UHCI init on dev 00:01.2 (io=c280)
(XEN) HVM13: Found 1 lpt ports
(XEN) HVM13: Found 1 serial ports
(XEN) HVM13: ATA controller 1 at 1f0/3f4/c2a0 (irq 14 dev 9)
(XEN) HVM13: ATA controller 2 at 170/374/c2a8 (irq 15 dev 9)
(XEN) HVM13: ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (20480 MiBytes)
(XEN) HVM13: Searching bootorder for: /pci@i0cf8/*@1,1/drive@0/disk@0
(XEN) HVM13: DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD]
(XEN) HVM13: Searching bootorder for: /pci@i0cf8/*@1,1/drive@1/disk@0
(XEN) HVM13: PS2 keyboard initialized
(XEN) HVM13: All threads complete.
(XEN) HVM13: Scan for option roms
(XEN) HVM13: Running option rom at c900:0003
(XEN) HVM13: pmm call arg1=1
(XEN) HVM13: pmm call arg1=0
(XEN) HVM13: pmm call arg1=1
(XEN) HVM13: pmm call arg1=0
(XEN) HVM13: Searching bootorder for: /pci@i0cf8/*@4
(XEN) HVM13: Press F12 for boot menu.
(XEN) HVM13:
(XEN) HVM13: drive 0x000fd930: PCHS=16383/16/63 translation=lba
LCHS=1024/255/63 s=41943040
(XEN) HVM13:
(XEN) HVM13: Space available for UMB: 000ca000-000ee800
(XEN) HVM13: Returned 61440 bytes of ZoneHigh
(XEN) HVM13: e820 map has 7 items:
(XEN) HVM13:   0: 0000000000000000 - 000000000009fc00 = 1 RAM
(XEN) HVM13:   1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
(XEN) HVM13:   2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
(XEN) HVM13:   3: 0000000000100000 - 00000000dffff000 = 1 RAM
(XEN) HVM13:   4: 00000000dffff000 - 00000000e0000000 = 2 RESERVED
(XEN) HVM13:   5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
(XEN) HVM13:   6: 0000000100000000 - 000000011f800000 = 1 RAM
(XEN) HVM13: enter handle_19:
(XEN) HVM13:   NULL
(XEN) HVM13: Booting from Hard Disk...
(XEN) HVM13: Booting from 0000:7c00
(XEN) stdvga.c:151:d13 leaving stdvga
(XEN) stdvga.c:147:d13 entering stdvga and caching modes
MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU uses below code to init 
RAM in xen_ram_init:

    ...
    block_len = ram_size;
    if (ram_size >= HVM_BELOW_4G_RAM_END) {
/* Xen does not allocate the memory continuously, and keep a hole at 
         * HVM_BELOW_4G_MMIO_START of HVM_BELOW_4G_MMIO_LENGTH
         */
        block_len += HVM_BELOW_4G_MMIO_LENGTH;
    }
    memory_region_init_ram(&ram_memory, "xen.ram", block_len);
    vmstate_register_ram_global(&ram_memory);

    if (ram_size >= HVM_BELOW_4G_RAM_END) {
        above_4g_mem_size = ram_size - HVM_BELOW_4G_RAM_END;
        below_4g_mem_size = HVM_BELOW_4G_RAM_END;
    } else {
        below_4g_mem_size = ram_size;
    }
    ...

HVM_BELOW_4G_RAM_END is f0000000. If we change HVM_BELOW_4G_RAM_END
to e0000000,
Which it's consistent with hvmloader when assigning a GPU, and then
guest worked
for us. So we wondering that xen_ram_init in QEMU should be consistent with 
hvmloader.
In addition, we found QEMU uses hardcode 0xe0000000 in pc_init1() as below. 
Should keep these places handle the consistent mmio hole or not?

    if (ram_size >= 0xe0000000 ) {
        above_4g_mem_size = ram_size - 0xe0000000;
        below_4g_mem_size = 0xe0000000;
    } else {
        above_4g_mem_size = 0;
        below_4g_mem_size = ram_size;
    }

--weidong
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


Hello,
What can of bug we can encounter by hardcoding the HVM_BELOW_4G_RAM_END to e0000000 ?
By doing this I'm able to boot on Win7 with gpu passthrough of an nvidia 560TI an 12Go of ram. 

Regards and thanks for all your work !

Youenn.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel

 

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.

Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.