Re: [Xen-devel] GPU passthrough issue when VM is configured with 4G memory

[Stefano Stabellini <stefano.stabellini@xxxxxxxxxxxxx>]

On Mon, 3 Jun 2013, Konrad Rzeszutek Wilk wrote:
> On Wed, May 29, 2013 at 05:18:24PM +0100, Stefano Stabellini wrote:
> > On Thu, 25 Apr 2013, Hanweidong wrote:
> > > > -----Original Message-----
> > > > From: xen-devel-bounces@xxxxxxxxxxxxx [mailto:xen-devel-
> > > > bounces@xxxxxxxxxxxxx] On Behalf Of Hanweidong
> > > > Sent: 2013å3æ26æ 17:38
> > > > To: Stefano Stabellini
> > > > Cc: George Dunlap; xudong.hao@xxxxxxxxx; Yanqiangjun; Luonengjun;
> > > > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
> > > > devel@xxxxxxxxxxxxx; xiantao.zhang@xxxxxxxxx
> > > > Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> > > > with 4G memory
> > > > 
> > > > 
> > > > > -----Original Message-----
> > > > > From: Stefano Stabellini [mailto:stefano.stabellini@xxxxxxxxxxxxx]
> > > > > Sent: 2013å3æ18æ 20:02
> > > > > To: Hanweidong
> > > > > Cc: George Dunlap; Stefano Stabellini; Yanqiangjun; Luonengjun;
> > > > > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
> > > > > devel@xxxxxxxxxxxxx; xudong.hao@xxxxxxxxx; xiantao.zhang@xxxxxxxxx
> > > > > Subject: RE: [Xen-devel] GPU passthrough issue when VM is configured
> > > > > with 4G memory
> > > > >
> > > > > On Wed, 13 Mar 2013, Hanweidong wrote:
> > > > > > MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU uses below
> > > > > code to init
> > > > > > RAM in xen_ram_init:
> > > > > >
> > > > > >     ...
> > > > > >     block_len = ram_size;
> > > > > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> > > > > >         /* Xen does not allocate the memory continuously, and keep
> > > > a
> > > > > hole at
> > > > > >          * HVM_BELOW_4G_MMIO_START of HVM_BELOW_4G_MMIO_LENGTH
> > > > > >          */
> > > > > >         block_len += HVM_BELOW_4G_MMIO_LENGTH;
> > > > > >     }
> > > > > >     memory_region_init_ram(&ram_memory, "xen.ram", block_len);
> > > > > >     vmstate_register_ram_global(&ram_memory);
> > > > > >
> > > > > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> > > > > >         above_4g_mem_size = ram_size - HVM_BELOW_4G_RAM_END;
> > > > > >         below_4g_mem_size = HVM_BELOW_4G_RAM_END;
> > > > > >     } else {
> > > > > >         below_4g_mem_size = ram_size;
> > > > > >     }
> > > > > >     ...
> > > > > >
> > > > > > HVM_BELOW_4G_RAM_END is f0000000. If we change HVM_BELOW_4G_RAM_END
> > > > > to e0000000,
> > > > > > Which it's consistent with hvmloader when assigning a GPU, and then
> > > > > guest worked
> > > > > > for us. So we wondering that xen_ram_init in QEMU should be
> > > > > consistent with
> > > > > > hvmloader.
> > > > > >
> > > > > > In addition, we found QEMU uses hardcode 0xe0000000 in pc_init1()
> > > > as
> > > > > below.
> > > > > > Should keep these places handle the consistent mmio hole or not?
> > > > > >
> > > > > >     if (ram_size >= 0xe0000000 ) {
> > > > > >         above_4g_mem_size = ram_size - 0xe0000000;
> > > > > >         below_4g_mem_size = 0xe0000000;
> > > > > >     } else {
> > > > > >         above_4g_mem_size = 0;
> > > > > >         below_4g_mem_size = ram_size;
> > > > > >     }
> > > > >
> > > > > The guys at Intel sent a couple of patches recently to fix this issue:
> > > > >
> > > > > http://marc.info/?l=xen-devel&m=136150317011027
> > > > > http://marc.info/?l=qemu-devel&m=136177475215360&w=2
> > > > >
> > > > > Do they solve your problem?
> > > > 
> > > > These two patches didn't solve our problem.
> > > > 
> > > 
> > > I debugged this issue with above two patches. I want to share some 
> > > information and discuss solution here. This issue is actually caused by 
> > > that a VM has a large pci hole (mmio size) which results in QEMU sets 
> > > memory regions inconsistently with hvmloader (QEMU uses hardcode 
> > > 0xe0000000 in pc_init1 and xen_ram_init). I created a virtual device with 
> > > 1GB mmio size to debug this issue. Firstly, QEMU set memory regions 
> > > except pci hole region in pc_init1() and xen_ram_init(), then hvmloader 
> > > calculated pci_mem_start as 0x80000000, and wrote it to TOM register, 
> > > which triggered QEMU to update pci hole region with 0x80000000 using 
> > > i440fx_update_pci_mem_hole(). Finally the windows 7 VM (configured 8G) 
> > > crashed with BSOD code 0x00000024. If I hardcode in QEMU pc_init1 and 
> > > xen_ram_init to match hvmloader's. Then the problem was gone. 
> > > 
> > > Althrough above two patches will pass actual pci hole start address to 
> > > QEMU, but it's too late, QEMU pc_init1() and xen_ram_init() already set 
> > > the other memory regions, and obviously the pci hole might overlap with 
> > > ram regions in this case. So I think hvmloader should setup pci devices 
> > > and calculate pci hole first, then QEMU can map memory regions correctly 
> > > from the beginning.  
> > > 
> > 
> > Thank you very much for your detailed analysis of the problem.
> > 
> > After reading this, I wonder how is possible that qemu-xen-traditional
> > does not have this issue, considering that AFAIK there is no way for
> > hvmloader to tell qemu-xen-traditional where the PCI hole starts.
> > 
> > The only difference between upstream QEMU and qemu-xen-traditional is
> > that the former would start the PCI hole at 0xf0000000 while the latter
> > would start the PCI hole at 0xe0000000.
> > 
> > So I would expect that your test, where hvmloader is updating the PCI
> > hole region to start at 0x80000000, would fail on qemu-xen-traditional
> > too.
> > 
> > Of course having the PCI hole starting unconditionally at 0xf0000000
> > makes it much easier to run into problems than starting it at
> > 0xe0000000.
> > 
> > 
> > Assuming that everything above is correct, this is what I would do:
> > 
> > 1) modify upstream QEMU to start the PCI hole at 0xe0000000, to match
> > qemu-xen-unstable in terms of configuration and not to introduce any
> > regressions. Do this for the Xen 4.3 release.
> > 
> > 2) for Xen 4.4 rework the two patches above and improve
> > i440fx_update_pci_mem_hole: resizing the pci_hole subregion is not
> > enough, it also needs to be able to resize the system memory region
> > (xen.ram) to make room for the bigger pci_hole
> 
> 
> Would that make migration more difficult - meaning if you have now two
> different QEMU versions where the PCI hole is different on them? Or is
> that not an issue and QEMU handles setting the layout nicely? Or is
> the 0xe0000000 the norm in Xen 4.1, and Xen 4.2?
>
> I am assuming you unplug the PCI device before you migrate of course.


the change in configuration is only for qemu-xen and upstream QEMU and
Xen 4.3 is the first release that defaults to it, so I don't think we
need to maintain save/restore compatibility yet. But from the next one
is going to be unavoidable.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel