Re: PCI devices passthrough on Arm design proposal

[Rahul Singh <Rahul.Singh@xxxxxxx>]

> On 18 Jul 2020, at 12:08 pm, Julien Grall <julien@xxxxxxx> wrote:
> 
> Hi,
> 
> On 17/07/2020 16:47, Bertrand Marquis wrote:
>>> On 17 Jul 2020, at 17:26, Julien Grall <julien@xxxxxxx> wrote:
>>> On 17/07/2020 15:47, Bertrand Marquis wrote:
>>>>>>>     pci=[ "PCI_SPEC_STRING", "PCI_SPEC_STRING", ...]
>>>>>>> 
>>>>>>> Guest will be only able to access the assigned devices and see the 
>>>>>>> bridges. Guest will not be able to access or see the devices that are 
>>>>>>> no assigned to him.
>>>>>>> 
>>>>>>> Limitation:
>>>>>>> * As of now all the bridges in the PCI bus are seen by the guest on the 
>>>>>>> VPCI bus.
>>>>>> Why do you want to expose all the bridges to a guest? Does this mean 
>>>>>> that the BDF should always match between the host and the guest?
>>>> That’s not really something that we wanted but this was the easiest way to 
>>>> go.
>>>> As said in a previous mail we could build a VPCI bus with a completely 
>>>> different topology but I am not sure of the advantages this would have.
>>>> Do you see some reason to do this ?
>>> 
>>> Yes :):
>>>  1) If a platform has two host controllers (IIRC Thunder-X has it) then you 
>>> would need to expose two host controllers to your guest. I think this is 
>>> undesirable if your guest is only using a couple of PCI devices on each 
>>> host controllers.
>>>  2) In the case of migration (live or not), you may want to use a 
>>> difference PCI card on the target platform. So your BDF and bridges may be 
>>> different.
>>> 
>>> Therefore I think the virtual topology can be beneficial.
>> I would see a big advantage definitely to have only one VPCI bus per guest 
>> and put all devices in their independently of the hardware domain the device 
>> is on.
>> But this will probably make the VPCI BARs value computation a bit more 
>> complex as we might end up with no space on the guest physical map for it.
>> This might make the implementation a lot more complex.
> 
> I am not sure to understand your argument about the space... You should be 
> able to find out the size of each BARs, so you can size the MMIO window 
> correctly. This shouldn't add a lot of complexity.
> 
> I am not asking any implementation for this, but we need to make sure the 
> design can easily be extended for other use cases. In the case of server, we 
> will likely want to expose a single vPCI to the guest.

This is something we have to work on how to implement the virtual topology for 
the guest. 

> 
>>> 
>>>>>>    - Is there any memory access that can bypassed the IOMMU (e.g 
>>>>>> doorbell)?
>>>> This is still something to be investigated as part of the MSI 
>>>> implementation.
>>>> If you have any idea here, feel free to tell us.
>>> 
>>> My memory is a bit fuzzy here. I am sure that the doorbell can bypass the 
>>> IOMMU on some platform, but I also vaguely remember that accesses to the 
>>> PCI host controller memory window may also bypass the IOMMU. A good reading 
>>> might be [2].
>>> 
>>> IIRC, I came to the conclusion that we may want to use the host memory map 
>>> in the guest when using the PCI passthrough. But maybe not on all the 
>>> platforms.
>> Definitely a lot of this would be easier if could use 1:1 mapping.
>> We will keep that in mind when we will start to investigate on the MSI part.
> 
> Hmmm... Maybe I wasn't clear enough but the problem is not only happening 
> with MSIs doorbells. It is also with the P2P transactions.
> 
> Again, I am not asking to implement it at the beginning. However, it would be 
> good to outline the potential limitations of the approach in your design.

As Bertrand mention once we start investigating on the MSI support we will have 
this in mind to proceed.
> 
> Cheers,
> 
> 
> -- 
> Julien Grall