Re: [Xen-devel] RESEND [Xen-unstable][Qemu-xen] HVM Guest reading of Expansion ROM from passthroughed PCI device returns data from emulated VGA rom

[Sander Eikelenboom <linux@xxxxxxxxxxxxxx>]

Monday, December 2, 2013, 9:39:37 PM, you wrote:

> On Mon, Dec 02, 2013 at 08:55:57PM +0100, Sander Eikelenboom wrote:
>> 
>> Monday, December 2, 2013, 8:41:31 PM, you wrote:
>> 
>> > On Tue, Sep 17, 2013 at 04:33:39PM +0200, Sander Eikelenboom wrote:
>> >> *RESEND* due to exceeding the mailinglists limit for attachment size.
>> >> 
>> >> Hi,
>> >> 
>> >> I'm trying to get secondary vga-passthrough on a HVM guest to work with a 
>> >> AMD HD6570 and the native kernel radeon driver and kernel modesetting.
>> >> So the guest still gets the emulated stdvga or cirrus device(used in my 
>> >> case here) as primary/boot vga adapter.
>> >> 
>> >> - When i don't passthrough the radeon card, the linux native radeon 
>> >> driver loads fine.
>> >> - When i do passtrough the device to a HVM with the same kernel:
>> >>   The driver in the guest tries to read the pci expansion rom from the 
>> >> passthroughed device to get the vbios.
>> >>   The driver reports a successful read, but fails because it can't find 
>> >> the right string at the right offset.
>> >> 
>> >> So I inspected the rom by using sysfs with:
>> >> echo 1 > /sys/bus/pci/devices/<BDF>/rom
>> >> cat /sys/bus/pci/devices/<BDF>/rom
>> >> 
>> >> - When i use this in dom0 (so without passthrough) i the contents of the 
>> >> ROM are valid (as expected since the driver loads fine)
>> >> - When i use this in the Guest (passthrouhed), the contents of the ROM i 
>> >> get are not from the passedthrough adapter, but from the emulated cirrus 
>> >> card.
>> >>   (it's the same as the 
>> >> "tools/firmware/vgabios/VGABIOS-lgpl-latest.cirrus.bin")
>> 
>> > I hadn't tried that, but I do get the same error:
>> 
>> > [    4.143445] [drm] radeon defaulting to kernel modesetting.
>> > [    4.143452] [drm] radeon kernel modesetting enabled.
>> > [    4.143525] checking generic (f0000000 160000) vs hw (110000000 
>> > 10000000)
>> > [    4.143976] xen: --> pirq=24 -> irq=40 (gsi=40)
>> > [    4.147760] [drm] initializing kernel modesetting (RV770 0x1002:0x9440 
>> > 0x174B:0xE850).
>> > [    4.148204] [drm] register mmio base: 0xF3040000
>> > [    4.148209] [drm] register mmio size: 65536
>> > [    4.152035] radeon 0000:00:06.0: >Expecting atombios for R600 GPU
>> > [    4.152040] radeon 0000:00:06.0: >Fatal error during GPU init 
>> > [    4.152044] [drm] radeon: finishing device.
>> > [    4.152047] [TTM] Memory type 2 has not been initialized
>> > [    4.168163] radeon 0000:00:06.0: >no bo for sa manager
>> > [    4.174380] radeon: probe of 0000:00:06.0 failed with error -22
>> 
>> > This is with the latest Xen version. No idea yet what is broken.
>> 
>> Nope i haven't been able to solve it so far.
>> Last weekend i tried vga passthrough with KVM and virtio and that worked 
>> without much hassle.
>> (although i cheated a bit and used the "rom_file" option to let qemu load 
>> the vga bios instead of loading it directly from the card)

> Aha :-)
>> The problem is many parts seems to be involved (hvmloader -> seabios -> qemu 
>> -> xen parts of qemu .. and xen as the hypervisor)
>> 
>> Qemu sets up the rombar, but there seems to be no backing from Xen, i also 
>> don't understand al the special casing around the rombar,
>> (when you leave out the exception of loading it from while, what is so 
>> different from a normal IOMEM region apart from that it should be readonly ?)

> If my memory serves me right - the only difference is that there needs
> to be a mechanism to "upload" the firmware in hvmloader. That way when
> QEMU starts it has the firmware blobls in memory and can execute them.

There is a option rom facility which filters out on nic's and disk classes.
Tried to comment that out, but it still wouldn't work ..
so it needs something else as well i guess.

My latest experiments were trying to use the "romfile" loading interface that 
qemu has for pci devices
and let it work the same way as kvm does it. But didn't get things to work that 
way either yet.
One of the things that can through you off is all the special casing where 
iterations are sometimes done
for all bars and sometime for all bars - 1.

> But that actually is not needed for PCI passthrough devices as . well
> ,you _pass_ them in. So the ROM should be read from the device. And I
> recall somebody doing it with LSI SAS and they found a bug in the BIOS.
> Also somebody did it with QLogic FibreChannel card. Would have to dig
> up the email archives to find out.
>> The virtio driver seems to be somewhat ahead, it also seems handle quirks 
>> for some cards already.

> virtio or virt-manager (and in extension the libs that do all of the
> nasty bits of shuffling bits around, enabling/disabling extensions,
> etc?)

Sorry it's the vfio pci kernel module and qemu stuff ..

This does the trick for me with KVM:

The device is binded to pci-stub on boot

echo 0000:07:00.0 > /sys/bus/pci/devices/0000\:07\:00.0/driver/unbind
echo 1002 6759 > /sys/bus/pci/drivers/vfio-pci/new_id

echo 0000:07:00.1 > /sys/bus/pci/devices/0000\:07\:00.1/driver/unbind
echo 1002 aa90 > /sys/bus/pci/drivers/vfio-pci/new_id

/usr/local/bin/qemu-system-x86_64 -machine type=pc,accel=kvm -cpu host -smp 
2,sockets=1,cores=2 -hda /root/xbmc.dump -m 1024 -boot c -vnc 0.0.0.0:1 -k 
en-us \
-device vfio-pci,host=07:00.0,x-vga=on,rombar=0,romfile=/root/07rom.bin \
-vga none


Which BTW seems like a limited interface in my opinion since using devid's 
instead of BDF's to specify the device to bind to vfio-pci would probably give 
problems
when you have more than one of such device in your machine. So i think xen's 
pciback does a better job there.

Under Xen i have / used to have (haven't tried it again lately) vga passthrough 
working on a linux guest with the closed source fglrx drivers and on a windows 
guest with a certain version of the catalyst drivers.
But these are a hhhhhhhhhhhhuge pain in the ass to match up with any recent 
kernel version and it's a bit crash prone .. but when it works,
i have ran openCL benchmarks with nearly baremetal performance. Used it for 
some experiments with openCV (for image recognition) offloading calculations to 
the GPU as well.


>> Haven't had much time lately to experiment, but the test with KVM at least 
>> showed it should be possible with my card.

> /me nods.
>> 
>> --
>> Sander
>> 



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel