Re: [Xen-users] Xen IB

[Joseph Glanville <joseph.glanville@xxxxxxxxxxxxxx>]

Hi Steven,

Sorry I missed your post but I thought I would clarify what we have working, both for the benefit of the list and yourself.

We are currently not using SR-IOV in production, I did however build a test stack consisting of a Connext-2 card and an IOMMU enabled Intel server (Intel VT-d) and Xen.org Source 4.1.
This setup allowed me to create 7 VFs per port (this was a dual port card) for a total of 14 virtual adapters.
The beta package I was using was dated March 2011 and seemed to be stable, it also presents 100% virtualized IB adapters that can either be used directly or passed through to virtual machines.
As the VFs appear on the PCI bus you must have a server supporting an IOMMU (most recent servers).
You can then use the standard OFED 1.5.X packages within the virtual machines to access the IB fabric.

The reasons this hasn't been used in production basically fall on the following reasons.

1) The driver was still beta - though it seemed to be stable we never deploy anything that is not certified to be stable.

2) As far as I can see there isn't a security model in place that would allow us to use it in a multi-tenant environment. This might not be an issue for most users but we are a public IaaS platform.

3) Our current software stack isn't able to make use of IOMMU PCI pass-through (limitation of our stack - not the Mellanox hardware, we have since resolved this)

The performance is as native, the tooling is simple - standard PCI pass-through easy to do with both xl or legacy xm.

I am looking forward to the production release so I can employ SR-IOV on our command and control stack and into the future when enough security features are available to offer it to clients on our multi-tenant platform.

Joseph.

On 6 November 2011 05:19, Steven Timm <timm@xxxxxxxx> wrote:

Could anyone define what "support" means?
We bought Mellanox connect-x2 cards 1+ years  ago and
we are still waiting for the drivers which they promised
at that time.  The last we talked to them the only
supported drivers are for VMware and even those are not
using the SR-IOV feature.  What they are working on for KVM
and Xen according to them is just a driver that presents
the IB card to the VM as a big network pipe, not something
that is recognizable with any regular mellanox driver
or that can be used with regular IB MPI drivers.

I hope I'm wrong and someone has got it working somewhere
because we have a lot of cash sunk into these cards
but the latest we heard from mellanox is that they still have
both software and firmware issues to be resolved.

Steve Timm

On Sat, 5 Nov 2011, Joseph Glanville wrote:

Hi

Mellanox ConnectX-2 and ConnectX-3 cards support SR-IOV.
Not sure about other vendors.

Joseph.

On 3 November 2011 03:47, Nick Khamis <symack@xxxxxxxxx> wrote:

Hello Vivien,

Just out of curiosity, which IB devices actually support things like
pci-passthrough or SR-IOV.

Thanks in Advance,

Nick.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

--

------------------------------------------------------------------
Steven C. Timm, Ph.D  (630) 840-8525
timm@xxxxxxxx  http://home.fnal.gov/~timm/
Fermilab Computing Division, Scientific Computing Facilities,
Grid Facilities Department, FermiGrid Services Group, Group Leader.
Lead of FermiCloud project.

--

Founder | Director | VP Research

Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56 99 52 | Mobile: 0428 754 846

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users