RE: Re: [Xen-devel] Memory mapping for PEG/PCIe Graphics Passthrough to <any> DomU

[Tim Moore <timothy.moore@xxxxxxxxxxx>]

That looks great! (I wish I had a Q45 with embedded Gfx)

I`m not using Jean`s patch in this case and therefore i`m not getting as far as 
you, maybe I need to hack some of the patch into my xen-unstable? i`ve not even 
looked into what it does yet.

Are the other cards you tested with PCI Express or just PCI?

It's the switchover from the Cirrus/vnc-vga to the pciback card that is 
failing, or should they be able to co-exist? 
My assumption is the pciback card bios needs to be reloaded to remap the memory 
adjacent to the Cirrus, but this is not happening.


-----Original Message-----
From: Alexia Benington [mailto:alexbenington@xxxxxxxxx] 
Sent: 30 March 2009 16:54
To: Tim Moore
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: Re: Re: [Xen-devel] Memory mapping for PEG/PCIe Graphics Passthrough 
to <any> DomU

Hi Tim,

Apologies for the double post, I think I didn't get the screenshot
through due to its size.

It depends on your definition of success. If it helps, I've attached a
screenshot of DomU's device manager. Dom0 will, obviously, lose access
to the display, when the gfx is handed to pciback. I'll be glad to
describe further if you require.

With nVidia Quadro, the driver sits in an idling loop. According to
the bugcheck error code, it's waiting for some kind of response from
the gfx. I suppose this reflects Jean's point that some drivers are
not very compatible.

I was mostly trying Jean's patch, which he released sometime last
year, into Xen unstable. I'm also curious how some have mentioned that
they were able to get vga passthrough to work correctly without
applying the patch. It might mostly work, but you'll still get the
emulated VGA in DomU.

-Alex

On Mon, Mar 30, 2009 at 11:33 AM, Tim Moore <timothy.moore@xxxxxxxxxxx> wrote:
>
> I agree, if any I was expecting the Nvidia card to work the most ..
>
> One interesting thing is that there is a fan on the Nvidia that spins down 
> when the bios is loaded, with Windows it is evident when the bios is 
> reset/loaded - just before the GUI is displayed.
> When booting Xen the fan stays high / fast as the pciback is loaded to 
> prevent Dom0 taking ownership.. boot the DomU (WinXP) and the fan spins down 
> the same as booting a native Win32 host .. aka I assume Bios/Card is 
> loaded/Reset here ..
>
> Pls can you explain FLR's ?
>
>
> -----Original Message-----
> From: Jean Guyader [mailto:jean.guyader@xxxxxxxxx]
> Sent: 30 March 2009 15:57
> To: Tim Moore
> Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
> Subject: Re: Re: [Xen-devel] Memory mapping for PEG/PCIe Graphics Passthrough 
> to <any> DomU
>
> Hi Tim,
>
> In the native case there is only one mapping, there isn't any
> remapping ones the bios has been executed. It is different in the hvm
> scenario.
>
> Anyway, the nvidia graphic card should works fine as a secondary, side
> by side with the cirrus. I don't know if this card support FLR's but
> that would be good to have a look.
>
> Jean
>
> 2009/3/30 Tim Moore <timothy.moore@xxxxxxxxxxx>:
>>
>> Hi Jean,
>>
>> I can understand the issue with the graphics card drivers, and I can see how 
>> that could be able problem. I am using a very modern nvidia and ati card for 
>> testing, Alex is also using a couple of other different varieties so we are 
>> able to test with a few different scenarios...
>>
>> How would this work under an MS kernel? these manufacturers do support dual 
>> cards and I have also tested my ATI/Nvidia host with a Windows OS and both 
>> cards are loaded and work fine. This says to me that the drivers can deal 
>> with PCI remapping of some sorts?
>>
>> Cheers,
>> Tim
>>
>> -----Original Message-----
>> From: Jean Guyader [mailto:jean.guyader@xxxxxxxxx]
>> Sent: 30 March 2009 15:38
>> To: Tim Moore
>> Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
>> Subject: Re: Re: [Xen-devel] Memory mapping for PEG/PCIe Graphics 
>> Passthrough to <any> DomU
>>
>> Hi Tim,
>>
>> Most of the time the problam doesn't come from the VT-d code but from
>> the graphic card device drivers. Most of them doesn't really like PCI
>> Bar remapping and try to use the old one.
>> For instance the intel igfx has never been designed to be used as a
>> secondary display adapter or even with another Host bridge than the
>> one on the motherboard.
>>
>> Jean
>>
>> 2009/3/30 Tim Moore <timothy.moore@xxxxxxxxxxx>:
>>> Hi Alex,
>>>
>>>
>>>
>>> When you say you had success with the onboard igfx, how much success? Have
>>> you ever had a passthrough VGA display from a DomU ?
>>>
>>>
>>>
>>> I have also seen the write-up too although a lot of work has been done on
>>> the VT-d support lately and I`m hoping to raise this again on the developer
>>> list and start to document some of the problems/issues so as we can find a
>>> way do solve it !
>>>
>>>
>>>
>>> Thanks for the reply! It still leaves this issue unsolved though ;(
>>>
>>>
>>>
>>> Tim
>>>
>>>
>>>
>>> ________________________________
>>>
>>> Subject: Re: [Xen-devel] Memory mapping for PEG/PCIe Graphics Passthrough to
>>>
>>>         <any> DomU
>>>
>>> From: Alexia Benington <alexbenington@xxxxxxxxx>
>>>
>>> To: Tim Moore <timothy.moore@xxxxxxxxxxx>
>>>
>>> Cc: "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> I've been trying to get this to work for some months, albeit not
>>>
>>> full-time. I concur that many would like to see it working. Some have
>>>
>>> posted that they managed to get it to work correctly, however, they
>>>
>>> almost always never reply to queries on how they got it to work, or
>>>
>>> their observations.
>>>
>>>
>>>
>>> I did manage to get rid of the yellow exclamation mark. You'll need to
>>>
>>> disable the emulated VGA bios, pretty much as described here
>>>
>>> http://lists.xensource.com/archives/html/xen-devel/2008-12/msg00474.html.
>>>
>>>
>>>
>>> Even after doing so, some of the registers are not directly passed
>>>
>>> through. See http://lists.xensource.com/archives/html/xen-devel/2009-02/msg=
>>>
>>> 01131.html.
>>>
>>> I'm not sure if that may affect anything.
>>>
>>>
>>>
>>> I've experimented with an Intel DQ45CB, nVidia Quadro NVS 290, nVidia
>>>
>>> GF9500, ATI Radion HD 2600XT. The best results I get are when using
>>>
>>> the onboard igfx as the primary, which is also being passed through.
>>>
>>>
>>>
>>> There is also a writeup which you might want to take a look
>>>
>>> http://staff.science.uva.nl/~delaat/sne-2008-2009/p22/report.pdf.
>>>
>>>
>>>
>>> I'm also trying to get DomU's output to the display.
>>>
>>>
>>>
>>> -Alex
>>>
>>>
>>>
>>> 2009/3/30 Tim Moore <timothy.moore@xxxxxxxxxxx>:
>>>
>>>> Hi
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> I have been testing xen-unstable.hg (29/03/09) over the past few days ..
>>>
>>>> trying to enable passthrough for VGA.
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> I am using Debian 5.0 (lenny) x64 and compiling from source each time. Tr=
>>>
>>> ied
>>>
>>>> both linux-2.6.18.8 and linux-2.6.27.5 from Xenbits =85
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> Hardware is Nvidia Geforce 8800 512mb GTS PCIe on Intel X58 (ASUS P6T
>>>
>>>> Deluxe) with Core i7 920 CPU.
>>>
>>>>
>>>
>>>> (had to hack the xen/drviers/pci/passthrough/vtd/dmar.c (line 388) and
>>>
>>>> remove the DMAR bail because of bad RMRR tables.
>>>
>>>>
>>>
>>>> I also have ATI X300 PCIe for Dom0 Console.
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> Using Xen parameter: iommu=3Dpassthrough,pv
>>>
>>>>
>>>
>>>> Using Dom) parameter: pciback.hide=3D(=91=85=92)
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> I compiled pciback as embedded in the kernel and I have tried many differ=
>>>
>>> ent
>>>
>>>> scenarios:
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> 1) Nvidia Primary (hidden with pciback.hide=3D kernel param) (no ATI card
>>>
>>>> installed)
>>>
>>>>
>>>
>>>> 2) ATI Primary, with Nvidia hidden (with pciback.hide=3D kernel param)
>>>
>>>>
>>>
>>>> 3) all scenarios with pciback as module and using echo devn >
>>>
>>>> /sys/bus/pci/drivers/pciback/new_slot and bind
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> In all scenarios I can successfully see the correct PCI device in the
>>>
>>>> DomU(s), I have tried primarily Windows XP 32bit SP3 and also:
>>>
>>>> linux-2.6.18.8 and linux-2.6.27.5 PV and non PV kernels as DomU and
>>>
>>>> (although I didn=92t load a driver) the lspci =96vvv showed the devices.
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> Now, I progressed to installing drivers under the Win32 DomU against the
>>>
>>>> Nvidia card, windows correctly identified the device and installed the
>>>
>>>> driver! (latest from nvidia.com). As we all know =96 Windows wanted a reb=
>>>
>>> oot-
>>>
>>>> beforehand a quick check of the device and it has a Yellow exclamation
>>>
>>>> indicating reboot required and resources unavailable.
>>>
>>>>
>>>
>>>> Reboot and XP starts up =85 BSOD .. nv4 driver fails .. L
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> Tried with the ATI X300 Primary (no NVidia connected) and went through th=
>>>
>>> e
>>>
>>>> same steps. Windows XP now boots without crash although the ATI device st=
>>>
>>> ill
>>>
>>>> shows a yellow exclamation and insufficient resources.
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> To summarise, pciback functions as expected for PCIe in Dom0 and DomU see=
>>>
>>> s
>>>
>>>> the device. All tools work fine (xm pci-attach / pci-detach / pci-list) a=
>>>
>>> nd
>>>
>>>> devid show and are managed correctly.
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> I have read all that there is on this subject and have come to the
>>>
>>>> conclusion that this is a problem with qemu-dm and the memory mapping for
>>>
>>>> the new Video device, the Cirrus or stdvga card is always present which I
>>>
>>>> believe may be causing the problem. I have tried nographics=3D1 but qemu-=
>>>
>>> dm
>>>
>>>> still maps the Video BIOS.
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> Then I noticed that qemu-dm has the new options for =93-vga none=94 .. so=
>>>
>>>  I
>>>
>>>> built a wrapper script for qemu-dm to launch with this param. Unfortuntat=
>>>
>>> ely
>>>
>>>> the DomU doesn=92t startup and the qemu log does not show any mention of =
>>>
>>> the
>>>
>>>> Video memory ! so I believe the switch works, but the DomU then fails to
>>>
>>>> boot L
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> To solve this and make it usable I think some development of qemu with
>>>
>>>> regards to the Cirrus VGA and memory management when using more than 1 vg=
>>>
>>> a
>>>
>>>> controller is necessary in order to remap the Cirrus or secondary vga
>>>
>>>> adapters in memory correctly. (its also difficult to configure a Windows
>>>
>>>> machine for a second vga when the primary is disabled lol.)
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> Also, I have run strace using a qemu-dm wrapper (very verbose) and analys=
>>>
>>> ed
>>>
>>>> the results =96nothing seems obvious, but i`m not an expert by far =85.
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> Windows has been able to support multiple VGA cards for a long time; I
>>>
>>>> assume that MS is remapping these somehow?
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> I have read in 2 different places that this has worked for others before
>>>
>>>> (without the legacy patch for igfx) and with xen-unstable.hg .. why does =
>>>
>>> it
>>>
>>>> not work for me ?
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> Any help / direction to further debug would be helpful .. I think many
>>>
>>>> people would like to get this one working once and for all!
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> Tim
>>>
>>>>
>>>
>>>> _______________________________________________
>>>
>>>> Xen-devel mailing list
>>>
>>>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>>>
>>>> http://lists.xensource.com/xen-devel
>>>
>>>>
>>>
>>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>>> http://lists.xensource.com/xen-devel
>>>
>>>
>>
>>
>>
>> --
>> Jean Guyader
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>> http://lists.xensource.com/xen-devel
>>
>
>
>
> --
> Jean Guyader
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel
>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel