Xen project Mailing List

Case 1: VM was powered on 'before' PCI device (FPGA) was brought on-line

-------------------------------------------------------------------------------------------------------------

dm-command: hot insert pass-through pci dev

register_real_device: Assigning real physical device 01:00.0 ...

register_real_device: Error: couldn't locate device in libpci structures

Case 2: VM was powered on 'after' PCI device (FPGA) was brought on-line

----------------------------------------------------------------------------------------------------------

dm-command: hot insert pass-through pci dev

register_real_device: Assigning real physical device 01:00.0 ...

register_real_device: Enable MSI translation via per device option

register_real_device: Disable power management

pt_iomul_init: Error: pt_iomul_init can't open file /dev/xen/pci_iomul: No such file or directory: 0x1:0x0.0x0

pt_register_regions: IO region registered (size=0x00100000 base_addr=0xc6000000)

pt_register_regions: IO region registered (size=0x00100000 base_addr=0xc6100000)

pt_msi_setup: msi mapped with pirq 84 pci_intx: intx=1

register_real_device: Real physical device 01:00.0 registered successfuly!

So clearly, qemu-dm reported an error for case (1) and reported success for case (2).

The interesting thing is why did the Xen xm pci-attach command report success in both cases. Also, from Xen's perspective it appears that the afore-mentioned PCI device is assigned to the VM in both cases (as verified by using the xm commands to get the list of remaining assignable PCI devices and the list of PCI devices attached to the VM).

Something seems wrong here.

Any thoughts?

-Anil

Subject: Re: [Xen-users] Hotplugging FPGA to running VM

This is a proprietary FPGA device. Each control blade in our router chassis has one of these FPGAs and its primary function is to control certain dedicated hardware components on the blade that are responsible for establishing connectivity with the high-speed inter-blade fabric on the router. I am sorry, but I cannot divulge additional details regarding the FPGA.

Here is a more detailed description of my actual usage scenario.

1) The FPGA image must be downloaded onto the device before it can be enabled for use.

2) The VM in question is responsible for performing step (1), which means that the FPGA is not visible to Xen/Dom-0 when

the VM is powered on.

3) After the VM performs step (1), the FPGA is seen in Dom-0. I use Xen pci-hotplug to assign the device to the VM.

This operation completes successfully (as reported by the appropriate xl command).

4) However, the guest OS inside the VM cannot see the FPGA device after step (3).

5) If I shutdown the VM and re-start it, the guest OS inside the VM sees the FPGA device (and everything then proceeds normally).

The troublesome part here is that I need to shutdown and restart the VM. I would like to avoid this and rely on Xen pci-hotplug to make the device visible to the guest OS inside the VM.

Thanks.

-Anil

From: xen-users-bounces@xxxxxxxxxxxxx [mailto:xen-users-bounces@xxxxxxxxxxxxx] On Behalf Of cyberhawk001@xxxxxxxxx
Sent: Thursday, May 31, 2012 10:49 AM
To: xen-users@xxxxxxxxxxxxx
Subject: Re: [Xen-users] Hotplugging FPGA to running VM

Just out of curiosity, what kind of FPGA device is this? Is it an FPGA development board or something already assembled and functioning that contains an FPGA device(s)?

Well, just being curious... :)

Yes, the guest OS has PCI hotplug support. I am able to hotplug a variety of PCI devices to the VM. The only one that doesn't work is the FPGA device. Even for this one, the only time it doesn't work, is if the VM was powered-on prior to the FPGA device being brought on-line on the host. If the FPGA was present before the VM was powered-on everything works as expected.

It somehow seems like new devices that are introduced in the host, after a VM has been started on it, cannot be hotplugged into the VM.

Thanks.

-Anil

From: theubaz@xxxxxxxxx [mailto:theubaz@xxxxxxxxx] On Behalf Of John Sherwood
Sent: Wednesday, May 30, 2012 7:20 PM
To: Anil Rao
Cc: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-users] Hotplugging FPGA to running VM

Are you sure the guest OS has PCI hotplug support?

On Wed, May 30, 2012 at 10:14 PM, Anil Rao <anil.rao@xxxxxxxxxxxx> wrote:

Hi,

I am trying to assign a FPGA (which is a PCI device) to a running VM. The interesting thing here is that the FPGA is not present when the VM was powered-on. I.e. sometime after the VM was started, the FPGA was brought on-line on the host by downloading its image to the device.

So far my attempts at using Xen PCI-passthrough have not been successful. Although Xen reports that the pass-through operation was successful, which I have verified using the appropriate xl command, I am unable to see the FPGA from within the guest OS.

I have tried both manual rescanning of the PCI bus inside the guest as well as making use of the pciehp driver. The only way I can (reliably) see the FPGA from within the guest, is to shutdown and restart the VM and then redo the hotplug operation. I want to avoid a VM reboot and was wondering if this is at all possible and if so what do I need to do to realize it.

[The virtualization host is based on the default Xen setup that comes with SLES 11 SP2]

Thanks for any help on this matter.

-Anil

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users