Re: [Xen-users] Hotplugging FPGA to running VM

[Anil Rao <anil.rao@xxxxxxxxxxxx>]

Thanks, John, for your thoughts.

 

The problem is not about a particular PCI device. Rather, it seems to affect several different types of devices.

 

What I am seeing is this. If a device is added to the host "after" a VM has been started on it, any attempt to hotplug that device to the (already running VM) results in the following.

 

   From Xen's point of view the hotplug operation is successful. The guest OS in the VM, however, cannot see the device.

 

If the VM was started after the device was added to the host, I don't have a problem.

 

So, once again, this is not particular to a single device. It affects all devices if the condition mentioned above is true.

 

Thanks.

-Anil

---

From: theubaz@xxxxxxxxx [mailto:theubaz@xxxxxxxxx] On Behalf Of John Sherwood
Sent: Thursday, May 31, 2012 9:16 PM
To: Anil Rao
Cc: xen-users@xxxxxxxxxxxxx
Subject: Re: [Xen-users] Hotplugging FPGA to running VM

If you've never seen this issue before and can't reproduce it with 
any other PCI devices, I'd suggest trying the devel mailing list, but I'd first 
collect whatever logs you can that are salient to the issue beforehand. 
 I've never had issues with PCI hot plug that were limited to a single 
device though, so I'm certainly drawing a blank here on what it could 
be.

On Thu, May 31, 2012 at 8:06 PM, Anil Rao <anil.rao@xxxxxxxxxxxx> wrote:

Since I am new to the xen-users list, I was wondering if my question is more appropriate for the xen-devel list. Any thoughts?

 

Thanks.

-Anil

---

From: xen-users-bounces@xxxxxxxxxxxxx [mailto:xen-users-bounces@xxxxxxxxxxxxx] On Behalf Of Anil Rao
Sent: Thursday, May 31, 2012 11:25 AM
To: cyberhawk001@xxxxxxxxx; xen-users@xxxxxxxxxxxxx

Subject: Re: [Xen-users] Hotplugging FPGA to running VM

This is a proprietary FPGA device. Each control blade in our router chassis has one of these FPGAs and its primary function is to control certain dedicated hardware components on the blade that are responsible for establishing connectivity with the high-speed inter-blade fabric on the router. I am sorry, but I cannot divulge additional details regarding the FPGA.

 

Here is a more detailed description of my actual usage scenario.

 

1) The FPGA image must be downloaded onto the device before it can be enabled for use.

2) The VM in question is responsible for performing step (1), which means that the FPGA is not visible to Xen/Dom-0 when

    the VM is powered on.

3) After the VM performs step (1), the FPGA is seen in Dom-0. I use Xen pci-hotplug to assign the device to the VM.

    This operation completes successfully (as reported by the appropriate xl command).

4) However, the guest OS inside the VM cannot see the FPGA device after step (3).

5) If I shutdown the VM and re-start it, the guest OS inside the VM sees the FPGA device (and everything then proceeds normally).

 

The troublesome part here is that I need to shutdown and restart the VM. I would like to avoid this and rely on Xen pci-hotplug to make the device visible to the guest OS inside the VM.

 

Thanks.

-Anil

 

---

From: xen-users-bounces@xxxxxxxxxxxxx [mailto:xen-users-bounces@xxxxxxxxxxxxx] On Behalf Of cyberhawk001@xxxxxxxxx
Sent: Thursday, May 31, 2012 10:49 AM
To: xen-users@xxxxxxxxxxxxx
Subject: Re: [Xen-users] Hotplugging FPGA to running VM

Just out of curiosity, what kind of FPGA device is this? Is it an FPGA development board or something already assembled and functioning that contains an FPGA device(s)?

Well, just being curious... :)

Yes, the guest OS has PCI hotplug support. I am able to hotplug a variety of PCI devices to the VM. The only one that doesn't work is the FPGA device. Even for this one, the only time it doesn't work, is if the VM was powered-on prior to the FPGA device being brought on-line on the host. If the FPGA was present before the VM was powered-on everything works as expected.

 

It somehow seems like new devices that are introduced in the host, after a VM has been started on it, cannot be hotplugged into the VM.

 

Thanks.

-Anil

 

 

---

From: theubaz@xxxxxxxxx [mailto:theubaz@xxxxxxxxx] On Behalf Of John Sherwood
Sent: Wednesday, May 30, 2012 7:20 PM
To: Anil Rao
Cc: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-users] Hotplugging FPGA to running VM

Are you sure the guest OS has PCI hotplug support?

On Wed, May 30, 2012 at 10:14 PM, Anil Rao <anil.rao@xxxxxxxxxxxx> wrote:

Hi,

 

I am trying to assign a FPGA (which is a PCI device) to a running VM. The interesting thing here is that the FPGA is not present when the VM was powered-on. I.e. sometime after the VM was started, the FPGA was brought on-line on the host by downloading its image to the device.

 

So far my attempts at using Xen PCI-passthrough have not been successful. Although Xen reports that the pass-through operation was successful, which I have verified using the appropriate xl command, I am unable to see the FPGA from within the guest OS.

 

I have tried both manual rescanning of the PCI bus inside the guest as well as making use of the pciehp driver. The only way I can (reliably) see the FPGA from within the guest, is to shutdown and restart the VM and then redo the hotplug operation. I want to avoid a VM reboot and was wondering if this is at all possible and if so what do I need to do to realize it.

 

[The virtualization host is based on the default Xen setup that comes with SLES 11 SP2]

 

Thanks for any help on this matter.

 

-Anil

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users