Re: [Xen-devel] GPU passthrough issue when VM is configured with 4G memory

[Hanweidong <hanweidong@xxxxxxxxxx>]

> -----Original Message-----
> From: dunlapg@xxxxxxxxx [mailto:dunlapg@xxxxxxxxx] On Behalf Of George
> Dunlap
> Sent: 2013年3月12日 18:40
> To: Hanweidong
> Cc: Stefano Stabellini; Yanqiangjun; Luonengjun; Wangzhenguo;
> Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-devel@xxxxxxxxxxxxx
> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memory
> 
> On Tue, Mar 12, 2013 at 5:45 AM, Hanweidong <hanweidong@xxxxxxxxxx>
> wrote:
> > I don't think it's a pass-through issue. We also encountered similar
> > issue when using ivshmem device with 512M mmio. The problem is
> hvmloader
> > adjusted pci_mem_start according to actual mmio size of VM, but QEMU
> does
> > not know the actual mmio size of VM, it just uses the
> HVM_BELOW_4G_RAM_END
> > and HVM_BELOW_4G_MMIO_LENGTH in xen_ram_init. This results in memory
> layout
> > mismatch between hvmloader and QEMU. I think it needs to make QEMU
> know
> > actual mmio size, and then it can make the same memory layout as
> hvmloader.
> 
> So I'm not an expert in this codepath, but it was my understanding
> that resizing the MMIO hole is actually the job of the guest BIOS: it
> is supposed to query the PCI devices to find out how much memory they
> need, and then relocate memory as appropriate.  XenServer never had
> any problem with the MMIO hole not being big enough when passing cards
> to the guest; the only problem we've ever had (as far as I know) is
> the issue I mentioned before, where there was a bug in some pci bridge
> cards that breaks VT-d if the guest physical address overlaps the host
> physical address range of another device.
> 
> Can you maybe boot Linux in an HVM domain and do an lspci or look at
> Linux's e820 map, and check to see whether the BIOS has in fact
> relocated the guest memory, or whether the MMIO hole does indeed start
> only at HVM_BELOW_4G_RAM_END?
> 

We tried Linux guest, but it stopped at grub screen, cannot boot up. Posted 
"xl dmesg" output as below:

(XEN) HVM13: HVM Loader
(XEN) HVM13: Detected Xen v4.3-unstable
(XEN) HVM13: Xenbus rings @0xfeffc000, event channel 3
(XEN) HVM13: System requested SeaBIOS
(XEN) HVM13: CPU speed is 2500 MHz
(XEN) irq.c:270: Dom13 PCI link 0 changed 0 -> 5
(XEN) HVM13: PCI-ISA link 0 routed to IRQ5
(XEN) irq.c:270: Dom13 PCI link 1 changed 0 -> 10
(XEN) HVM13: PCI-ISA link 1 routed to IRQ10
(XEN) irq.c:270: Dom13 PCI link 2 changed 0 -> 11
(XEN) HVM13: PCI-ISA link 2 routed to IRQ11
(XEN) irq.c:270: Dom13 PCI link 3 changed 0 -> 5
(XEN) HVM13: PCI-ISA link 3 routed to IRQ5
(XEN) HVM13: pci dev 01:2 INTD->IRQ5
(XEN) HVM13: pci dev 01:3 INTA->IRQ10
(XEN) HVM13: pci dev 03:0 INTA->IRQ5
(XEN) HVM13: pci dev 04:0 INTA->IRQ5
(XEN) HVM13: pci dev 05:0 INTA->IRQ10
(XEN) HVM13: pci dev 05:0 bar 14 size lx: 08000000
(XEN) memory_map:add: dom13 gfn=e0000 mfn=e8000 nr=8000
(XEN) memory_map:add: dom13 gfn=e8000 mfn=f0000 nr=4000
(XEN) HVM13: pci dev 05:0 bar 1c size lx: 04000000
(XEN) HVM13: pci dev 02:0 bar 10 size lx: 02000000
(XEN) memory_map:add: dom13 gfn=ee000 mfn=bc000 nr=2000
(XEN) HVM13: pci dev 05:0 bar 10 size lx: 02000000
(XEN) HVM13: pci dev 03:0 bar 14 size lx: 01000000
(XEN) HVM13: pci dev 05:0 bar 30 size lx: 00080000
(XEN) HVM13: pci dev 02:0 bar 30 size lx: 00010000
(XEN) HVM13: pci dev 04:0 bar 30 size lx: 00010000
(XEN) HVM13: pci dev 02:0 bar 14 size lx: 00001000
(XEN) HVM13: pci dev 03:0 bar 10 size lx: 00000100
(XEN) HVM13: pci dev 04:0 bar 10 size lx: 00000100
(XEN) HVM13: pci dev 04:0 bar 14 size lx: 00000100
(XEN) HVM13: pci dev 05:0 bar 24 size lx: 00000080
(XEN) ioport_map:add: dom13 gport=c200 mport=2000 nr=80
(XEN) HVM13: pci dev 01:2 bar 20 size lx: 00000020
(XEN) HVM13: pci dev 01:1 bar 20 size lx: 00000010
(XEN) HVM13: Multiprocessor initialisation:
(XEN) HVM13:  - CPU0 ... 46-bit phys ... fixed MTRRs ... var MTRRs [3/8] ... 
done.
(XEN) HVM13: Testing HVM environment:
(XEN) HVM13:  - REP INSB across page boundaries ... passed
(XEN) HVM13:  - GS base MSRs and SWAPGS ... passed
(XEN) HVM13: Passed 2 of 2 tests
(XEN) HVM13: Writing SMBIOS tables ...
(XEN) HVM13: Loading SeaBIOS ...
(XEN) HVM13: Creating MP tables ...
(XEN) HVM13: Loading ACPI ...
(XEN) HVM13: vm86 TSS at fc00a000
(XEN) HVM13: BIOS map:
(XEN) HVM13:  10000-100d3: Scratch space
(XEN) HVM13:  e0000-fffff: Main BIOS
(XEN) HVM13: E820 table:
(XEN) HVM13:  [00]: 00000000:00000000 - 00000000:000a0000: RAM
(XEN) HVM13:  HOLE: 00000000:000a0000 - 00000000:000e0000
(XEN) HVM13:  [01]: 00000000:000e0000 - 00000000:00100000: RESERVED
(XEN) HVM13:  [02]: 00000000:00100000 - 00000000:e0000000: RAM
(XEN) HVM13:  HOLE: 00000000:e0000000 - 00000000:fc000000
(XEN) HVM13:  [03]: 00000000:fc000000 - 00000001:00000000: RESERVED
(XEN) HVM13:  [04]: 00000001:00000000 - 00000001:1f800000: RAM
(XEN) HVM13: Invoking SeaBIOS ...
(XEN) HVM13: SeaBIOS (version rel-1.7.1-0-g51755c3-20130227_164001-linux-DPMZPO)
(XEN) HVM13: 
(XEN) HVM13: Found Xen hypervisor signature at 40000000
(XEN) HVM13: xen: copy e820...
(XEN) HVM13: Ram Size=0xe0000000 (0x000000001f800000 high)
(XEN) HVM13: Relocating low data from 0x000e2490 to 0x000ef790 (size 2156)
(XEN) HVM13: Relocating init from 0x000e2cfc to 0xdffe20f0 (size 56804)
(XEN) HVM13: CPU Mhz=2501
(XEN) HVM13: Found 9 PCI devices (max PCI bus is 00)
(XEN) HVM13: Allocated Xen hypercall page at dffff000
(XEN) HVM13: Detected Xen v4.3-unstable
(XEN) HVM13: Found 1 cpu(s) max supported 1 cpu(s)
(XEN) HVM13: xen: copy BIOS tables...
(XEN) HVM13: Copying SMBIOS entry point from 0x00010010 to 0x000fdb10
(XEN) HVM13: Copying MPTABLE from 0xfc001140/fc001150 to 0x000fda30
(XEN) HVM13: Copying PIR from 0x00010030 to 0x000fd9b0
(XEN) HVM13: Copying ACPI RSDP from 0x000100b0 to 0x000fd980
(XEN) HVM13: Scan for VGA option rom
(XEN) HVM13: Running option rom at c000:0003
(XEN) stdvga.c:147:d13 entering stdvga and caching modes
(XEN) HVM13: Turning on vga text mode console
(XEN) HVM13: SeaBIOS (version rel-1.7.1-0-g51755c3-20130227_164001-linux-DPMZPO)
(XEN) HVM13: 
(XEN) HVM13: UHCI init on dev 00:01.2 (io=c280)
(XEN) HVM13: Found 1 lpt ports
(XEN) HVM13: Found 1 serial ports
(XEN) HVM13: ATA controller 1 at 1f0/3f4/c2a0 (irq 14 dev 9)
(XEN) HVM13: ATA controller 2 at 170/374/c2a8 (irq 15 dev 9)
(XEN) HVM13: ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (20480 MiBytes)
(XEN) HVM13: Searching bootorder for: /pci@i0cf8/*@1,1/drive@0/disk@0
(XEN) HVM13: DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD]
(XEN) HVM13: Searching bootorder for: /pci@i0cf8/*@1,1/drive@1/disk@0
(XEN) HVM13: PS2 keyboard initialized
(XEN) HVM13: All threads complete.
(XEN) HVM13: Scan for option roms
(XEN) HVM13: Running option rom at c900:0003
(XEN) HVM13: pmm call arg1=1
(XEN) HVM13: pmm call arg1=0
(XEN) HVM13: pmm call arg1=1
(XEN) HVM13: pmm call arg1=0
(XEN) HVM13: Searching bootorder for: /pci@i0cf8/*@4
(XEN) HVM13: Press F12 for boot menu.
(XEN) HVM13: 
(XEN) HVM13: drive 0x000fd930: PCHS=16383/16/63 translation=lba 
LCHS=1024/255/63 s=41943040
(XEN) HVM13: 
(XEN) HVM13: Space available for UMB: 000ca000-000ee800
(XEN) HVM13: Returned 61440 bytes of ZoneHigh
(XEN) HVM13: e820 map has 7 items:
(XEN) HVM13:   0: 0000000000000000 - 000000000009fc00 = 1 RAM
(XEN) HVM13:   1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
(XEN) HVM13:   2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
(XEN) HVM13:   3: 0000000000100000 - 00000000dffff000 = 1 RAM
(XEN) HVM13:   4: 00000000dffff000 - 00000000e0000000 = 2 RESERVED
(XEN) HVM13:   5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
(XEN) HVM13:   6: 0000000100000000 - 000000011f800000 = 1 RAM
(XEN) HVM13: enter handle_19:
(XEN) HVM13:   NULL
(XEN) HVM13: Booting from Hard Disk...
(XEN) HVM13: Booting from 0000:7c00
(XEN) stdvga.c:151:d13 leaving stdvga
(XEN) stdvga.c:147:d13 entering stdvga and caching modes

MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU uses below code to init 
RAM in xen_ram_init: 

    ...
    block_len = ram_size;
    if (ram_size >= HVM_BELOW_4G_RAM_END) {
        /* Xen does not allocate the memory continuously, and keep a hole at
         * HVM_BELOW_4G_MMIO_START of HVM_BELOW_4G_MMIO_LENGTH
         */
        block_len += HVM_BELOW_4G_MMIO_LENGTH;
    }
    memory_region_init_ram(&ram_memory, "xen.ram", block_len);
    vmstate_register_ram_global(&ram_memory);

    if (ram_size >= HVM_BELOW_4G_RAM_END) {
        above_4g_mem_size = ram_size - HVM_BELOW_4G_RAM_END;
        below_4g_mem_size = HVM_BELOW_4G_RAM_END;
    } else {
        below_4g_mem_size = ram_size;
    }
    ...

HVM_BELOW_4G_RAM_END is f0000000. If we change HVM_BELOW_4G_RAM_END to 
e0000000, 
Which it's consistent with hvmloader when assigning a GPU, and then guest 
worked 
for us. So we wondering that xen_ram_init in QEMU should be consistent with 
hvmloader. 

In addition, we found QEMU uses hardcode 0xe0000000 in pc_init1() as below. 
Should keep these places handle the consistent mmio hole or not?

    if (ram_size >= 0xe0000000 ) {
        above_4g_mem_size = ram_size - 0xe0000000;
        below_4g_mem_size = 0xe0000000;
    } else {
        above_4g_mem_size = 0;
        below_4g_mem_size = ram_size;
    }

--weidong
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel