Re: [Xen-devel] GPU passthrough issue when VM is configured with 4G memory

[Hanweidong <hanweidong@xxxxxxxxxx>]

> -----Original Message-----
> From: George Dunlap [mailto:george.dunlap@xxxxxxxxxxxxx]
> Sent: 2013年3月6日 20:43
> To: Hanweidong
> Cc: Gonglei (Arei); xen-devel@xxxxxxxxxxxxx; Yangxiaowei; Yanqiangjun;
> Luonengjun; Wangzhenguo
> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memory
> 
> On 06/03/13 11:38, Hanweidong wrote:
> >> -----Original Message-----
> >> From: dunlapg@xxxxxxxxx [mailto:dunlapg@xxxxxxxxx] On Behalf Of
> George
> >> Dunlap
> >> Sent: 2013年3月5日 20:59
> >> To: Gonglei (Arei)
> >> Cc: xen-devel@xxxxxxxxxxxxx; Yangxiaowei; Yanqiangjun; Luonengjun;
> >> Wangzhenguo; Hanweidong
> >> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> >> with 4G memory
> >>
> >> On Mon, Mar 4, 2013 at 8:10 AM, Gonglei (Arei)
> <arei.gonglei@xxxxxxxxxx>
> >> wrote:
> >>> Hi,all
> >>>
> >>> I have tried to passthrough GPU card(Nvidia quadro 4000) on the
> >> latest Xen
> >>> unstable version (QEMU is using Qemu-upsteam-unstable, not
> >> traditional
> >>> Qemu). This issue as below:
> >>>
> >>>        Windows7 64-bit guest will blue screen when GPU passthrough
> >> configure
> >>> 4g memory,blue screen code is 50, and SUSE 11 64-bit guest will
> >> always stay
> >>> at the grub screen.  I noticed that it will relocate RAM that
> >> overlaps PCI
> >>> space in pci_setup()(tools/hvmloader/pci.c). If VM memory is
> >> configured with
> >>> 3G, it won't cause relocate RAM that overlaps PCI space in
> >> pci_setup(), and
> >>> GPU pass-through is no problem. So it seems this issue is related
> to
> >>> "relocate RAM" in pci_setup().
> >> So one issue XenServer found with passing through GPUs is that there
> >> are bugs in some PCI bridges that completely break VT-d.  The issue
> >> was that if the *guest* physical address space overlapped the *host*
> >> physical address of a different device, that the PCI bridges would
> >> send traffic from the passed-through card intended for the guest to
> >> another card instead.  The work-around was to make the hole in the
> >> guest MMIO space the same size as the host MMIO hole.  I'm not sure
> if
> >> that made it upstream or not -- let me check...
> >>
> > Hi George,
> >
> > Could you post your patch and let us have a try with it? Thanks!
> 
> So the patch got checked in, but there still may be some more work if
> you want to use it. :-)
> 
> The patch modifies xc_hvm_build_args structure to include a field
> called
> "mmio_size". If this is set to zero, it will default to
> HVM_BELOW_4G_MMIO_LENGTH; otherwise, it will be the size of the default
> MMIO hole set during the build process. The guest BIOS may modify this
> at boot time to make it bigger, but it doesn't make it smaller.
> 
> Since this was designed for xapi, however, which calls libxc directly,
> we didn't add any options to xend / xl / libxl to set this option.
> 
> The easiest way to test it probably is just to hard-code
> HVM_BELOW_4G_MMIO_LENGTH to a new value (from the description, setting
> it to 1GiB should be sufficient).
> 
> Then if you want to use it in production, you probably want to either:
> 1. Try it with the latest version of XCP (which I think has an option
> you can set)
> 2. Implement a config option for xl that allows you to set the MMIO
> hole
> size.
> 
> #2 should be a relatively straightforward matter of "plumbing", and
> would be a welcome contribution. :-)
> 
> If you do implement #2, it might be nice to have an option of
> "mmio_hole_size=host", which will set the guest mmio hole to the same
> size as the host. That's what we implemented for XenServer, to make
> sure
> there would never be any collisions.
> 

We rooted caused this issue: in pci_setup, it relocates pci_mem_start from 
0xF0000000 to 0XE0000000 because GPU has big more MMIO, but in QEMU, 
xen_ram_init directly uses the macro HVM_BELOW_4G_RAM_END and 
HVM_BELOW_4G_MMIO_LENGTH, doesn't do corresponding relocation like pci_setup
does.

We tried to hardcod HVM_BELOW_4G_RAM_END to 0XE0000000, GPU pass-through
Worked.

--Weidong


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel