Re: [Xen-devel] VIRTIO - compatibility with different virtualization solutions

[Anthony Liguori <anthony@xxxxxxxxxxxxx>]

On Wed, Feb 19, 2014 at 5:31 PM, Rusty Russell <rusty@xxxxxxxxxxx> wrote:
> Anthony Liguori <anthony@xxxxxxxxxxxxx> writes:
>> On Tue, Feb 18, 2014 at 4:26 PM, Rusty Russell <rusty@xxxxxxxxxxx> wrote:
>>> Daniel Kiper <daniel.kiper@xxxxxxxxxx> writes:
>>>> Hi,
>>>>
>>>> Below you could find a summary of work in regards to VIRTIO compatibility 
>>>> with
>>>> different virtualization solutions. It was done mainly from Xen point of 
>>>> view
>>>> but results are quite generic and can be applied to wide spectrum
>>>> of virtualization platforms.
>>>
>>> Hi Daniel,
>>>
>>>         Sorry for the delayed response, I was pondering...  CC changed
>>> to virtio-dev.
>>>
>>> From a standard POV: It's possible to abstract out the where we use
>>> 'physical address' for 'address handle'.  It's also possible to define
>>> this per-platform (ie. Xen-PV vs everyone else).  This is sane, since
>>> Xen-PV is a distinct platform from x86.
>>
>> I'll go even further and say that "address handle" doesn't make sense too.
>
> I was trying to come up with a unique term, I wasn't trying to define
> semantics :)

Understood, that wasn't really directed at you.

> There are three debates here now: (1) what should the standard say, and

The standard should say, "physical address"

> (2) how would Linux implement it,

Linux should use the PCI DMA API.

> (3) should we use each platform's PCI
> IOMMU.

Just like any other PCI device :-)

>> Just using grant table references is not enough to make virtio work
>> well under Xen.  You really need to use bounce buffers ala persistent
>> grants.
>
> Wait, if you're using bounce buffers, you didn't make it "work well"!

Preaching to the choir man...  but bounce buffering is proven to be
faster than doing grant mappings on every request.  xen-blk does
bounce buffering by default and I suspect netfront is heading that
direction soon.

It would be a lot easier to simply have a global pool of grant tables
that effectively becomes the DMA pool.  Then the DMA API can bounce
into that pool and those addresses can be placed on the ring.

It's a little different for Xen because now the backends have to deal
with physical addresses but the concept is still the same.

>> I think what you ultimately want is virtio using a DMA API (I know
>> benh has scoffed at this but I don't buy his argument at face value)
>> and a DMA layer that bounces requests to a pool of persistent grants.
>
> We can have a virtio DMA API, sure.  It'd be a noop for non-Xen.
>
> But emulating the programming of an IOMMU seems masochistic.  PowerPC
> have made it clear they don't want this.

I don't think the argument is all that clear.  Wouldn't it be nice for
other PCI devices to be faster under Power KVM?  Why not have not
change the DMA API under Power Linux to detect that it's under KVM and
simply not make any hypercalls?

>  And noone else has come up
> with a compelling reason to want this: virtio passthrough?

So I can run Xen under QEMU and use virtio-blk and virtio-net as the
device model.  Xen PV uses the DMA API to do mfn -> pfn mapping and
since virtio doesn't use it, it's the only PCI device in the QEMU
device model that doesn't actually work when running Xen under QEMU.

Regards,

Anthony Liguori

>>> For platforms using EPT, I don't think you want anything but guest
>>> addresses, do you?
>>>
>>> From an implementation POV:
>>>
>>> On IOMMU, start here for previous Linux discussion:
>>>         
>>> http://thread.gmane.org/gmane.linux.kernel.virtualization/14410/focus=14650
>>>
>>> And this is the real problem.  We don't want to use the PCI IOMMU for
>>> PCI devices.  So it's not just a matter of using existing Linux APIs.
>>
>> Is there any data to back up that claim?
>
> Yes, for powerpc.  Implementer gets to measure, as always.  I suspect
> that if you emulate an IOMMU on Intel, your performance will suck too.
>
>> Just because power currently does hypercalls for anything that uses
>> the PCI IOMMU layer doesn't mean this cannot be changed.
>
> Does someone have an implementation of an IOMMU which doesn't use
> hypercalls, or is this theoretical?
>
>>  It's pretty
>> hacky that virtio-pci just happens to work well by accident on power
>> today.  Not all architectures have this limitation.
>
> It's a fundamental assumption of virtio that the host can access all of
> guest memory.  That's paravert, not a hack.
>
> But tomayto tomatoh aside, it's unclear to me how you'd build an
> efficient IOMMU today.  And it's unclear what benefit you'd gain.  But
> the cost for Power is clear.
>
> So if someone wants do to this for PCI, they need to implement it and
> benchmark.  But this is a little orthogonal to the Xen discussion.
>
> Cheers,
> Rusty.
>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel