Re: [Xen-devel] GPU passthrough issue when VM is configured with 4G memory

[Hanweidong <hanweidong@xxxxxxxxxx>]

> -----Original Message-----
> From: George Dunlap [mailto:george.dunlap@xxxxxxxxxxxxx]
> Sent: 2013年3月7日 18:16
> To: Hanweidong
> Cc: Gonglei (Arei); xen-devel@xxxxxxxxxxxxx; Yangxiaowei; Yanqiangjun;
> Luonengjun; Wangzhenguo; Anthony Perard; Stefano Stabellini
> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memory
> 
> On 07/03/13 07:51, Hanweidong wrote:
> >> -----Original Message-----
> >> From: George Dunlap [mailto:george.dunlap@xxxxxxxxxxxxx]
> >> Sent: 2013年3月6日 20:43
> >> To: Hanweidong
> >> Cc: Gonglei (Arei); xen-devel@xxxxxxxxxxxxx; Yangxiaowei;
> Yanqiangjun;
> >> Luonengjun; Wangzhenguo
> >> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> >> with 4G memory
> >>
> >> On 06/03/13 11:38, Hanweidong wrote:
> >>>> -----Original Message-----
> >>>> From: dunlapg@xxxxxxxxx [mailto:dunlapg@xxxxxxxxx] On Behalf Of
> >> George
> >>>> Dunlap
> >>>> Sent: 2013年3月5日 20:59
> >>>> To: Gonglei (Arei)
> >>>> Cc: xen-devel@xxxxxxxxxxxxx; Yangxiaowei; Yanqiangjun; Luonengjun;
> >>>> Wangzhenguo; Hanweidong
> >>>> Subject: Re: [Xen-devel] GPU passthrough issue when VM is
> configured
> >>>> with 4G memory
> >>>>
> >>>> On Mon, Mar 4, 2013 at 8:10 AM, Gonglei (Arei)
> >> <arei.gonglei@xxxxxxxxxx>
> >>>> wrote:
> >>>>> Hi,all
> >>>>>
> >>>>> I have tried to passthrough GPU card(Nvidia quadro 4000) on the
> >>>> latest Xen
> >>>>> unstable version (QEMU is using Qemu-upsteam-unstable, not
> >>>> traditional
> >>>>> Qemu). This issue as below:
> >>>>>
> >>>>>        Windows7 64-bit guest will blue screen when GPU
> passthrough
> >>>> configure
> >>>>> 4g memory,blue screen code is 50, and SUSE 11 64-bit guest will
> >>>> always stay
> >>>>> at the grub screen.  I noticed that it will relocate RAM that
> >>>> overlaps PCI
> >>>>> space in pci_setup()(tools/hvmloader/pci.c). If VM memory is
> >>>> configured with
> >>>>> 3G, it won't cause relocate RAM that overlaps PCI space in
> >>>> pci_setup(), and
> >>>>> GPU pass-through is no problem. So it seems this issue is related
> >> to
> >>>>> "relocate RAM" in pci_setup().
> >>>> So one issue XenServer found with passing through GPUs is that
> there
> >>>> are bugs in some PCI bridges that completely break VT-d.  The
> issue
> >>>> was that if the *guest* physical address space overlapped the
> *host*
> >>>> physical address of a different device, that the PCI bridges would
> >>>> send traffic from the passed-through card intended for the guest
> to
> >>>> another card instead.  The work-around was to make the hole in the
> >>>> guest MMIO space the same size as the host MMIO hole.  I'm not
> sure
> >> if
> >>>> that made it upstream or not -- let me check...
> >>>>
> >>> Hi George,
> >>>
> >>> Could you post your patch and let us have a try with it? Thanks!
> >> So the patch got checked in, but there still may be some more work
> if
> >> you want to use it. :-)
> >>
> >> The patch modifies xc_hvm_build_args structure to include a field
> >> called
> >> "mmio_size". If this is set to zero, it will default to
> >> HVM_BELOW_4G_MMIO_LENGTH; otherwise, it will be the size of the
> default
> >> MMIO hole set during the build process. The guest BIOS may modify
> this
> >> at boot time to make it bigger, but it doesn't make it smaller.
> >>
> >> Since this was designed for xapi, however, which calls libxc
> directly,
> >> we didn't add any options to xend / xl / libxl to set this option.
> >>
> >> The easiest way to test it probably is just to hard-code
> >> HVM_BELOW_4G_MMIO_LENGTH to a new value (from the description,
> setting
> >> it to 1GiB should be sufficient).
> >>
> >> Then if you want to use it in production, you probably want to
> either:
> >> 1. Try it with the latest version of XCP (which I think has an
> option
> >> you can set)
> >> 2. Implement a config option for xl that allows you to set the MMIO
> >> hole
> >> size.
> >>
> >> #2 should be a relatively straightforward matter of "plumbing", and
> >> would be a welcome contribution. :-)
> >>
> >> If you do implement #2, it might be nice to have an option of
> >> "mmio_hole_size=host", which will set the guest mmio hole to the
> same
> >> size as the host. That's what we implemented for XenServer, to make
> >> sure
> >> there would never be any collisions.
> >>
> > We rooted caused this issue: in pci_setup, it relocates pci_mem_start
> from
> > 0xF0000000 to 0XE0000000 because GPU has big more MMIO, but in QEMU,
> > xen_ram_init directly uses the macro HVM_BELOW_4G_RAM_END and
> > HVM_BELOW_4G_MMIO_LENGTH, doesn't do corresponding relocation like
> pci_setup
> > does.
> >
> > We tried to hardcod HVM_BELOW_4G_RAM_END to 0XE0000000, GPU pass-
> through
> > Worked.
> 
> I'm not super familiar with the HVM qemu / guest BIOS stuff, but it
> sounds like that's a bug in pass-through that needs to be sorted out.
> Anthony, can you comment?
> 

I don't think it's a pass-through issue. We also encountered similar 
issue when using ivshmem device with 512M mmio. The problem is hvmloader
adjusted pci_mem_start according to actual mmio size of VM, but QEMU does 
not know the actual mmio size of VM, it just uses the HVM_BELOW_4G_RAM_END 
and HVM_BELOW_4G_MMIO_LENGTH in xen_ram_init. This results in memory layout 
mismatch between hvmloader and QEMU. I think it needs to make QEMU know 
actual mmio size, and then it can make the same memory layout as hvmloader. 

--Weidong

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel