Re: [PATCH v10 13/17] vpci: add initial support for virtual PCI bus topology

[Julien Grall <julien@xxxxxxx>]

Hi Volodymyr,

On 17/11/2023 14:09, Volodymyr Babchuk wrote:
> Hi Stefano,
>
> Stefano Stabellini <sstabellini@xxxxxxxxxx> writes:
>
> > On Fri, 17 Nov 2023, Volodymyr Babchuk wrote:
> > > > I still think, no matter the BDF allocation scheme, that we should try
> > > > to avoid as much as possible to have two different PCI Root Complex
> > > > emulators. Ideally we would have only one PCI Root Complex emulated by
> > > > Xen. Having 2 PCI Root Complexes both of them emulated by Xen would be
> > > > tolerable but not ideal.
> > > But what is exactly wrong with this setup?
> > [...]
> >
> > > > The worst case I would like to avoid is to have
> > > > two PCI Root Complexes, one emulated by Xen and one emulated by QEMU.
> > > This is how our setup works right now.
> > If we have:
> > - a single PCI Root Complex emulated in Xen
> > - Xen is safety certified
> > - individual Virtio devices emulated by QEMU with grants for memory
> >
> > We can go very far in terms of being able to use Virtio in safety
> > use-cases. We might even be able to use Virtio (frontends) in a SafeOS.
> >
> > On the other hand if we put an additional Root Complex in QEMU:
> > - we pay a price in terms of complexity of the codebase
> > - we pay a price in terms of resource utilization
> > - we have one additional problem in terms of using this setup with a
> >    SafeOS (one more device emulated by a non-safe component)
> >
> > Having 2 PCI Root Complexes both emulated in Xen is a middle ground
> > solution because:
> > - we still pay a price in terms of resource utilization
> > - the code complexity goes up a bit but hopefully not by much
> > - there is no impact on safety compared to the ideal scenario
> >
> > This is why I wrote that it is tolerable.
> Ah, I see now. Yes, I am agree with this. Also I want to add some more
> points:
>
> - There is ongoing work on implementing virtio backends as a separate
>    applications, written in Rust. Linaro are doing this part. Right now
>    they are implementing only virtio-mmio, but if they want to provide
>    virtio-pci as well, they will need a mechanism to plug only
>    virtio-pci, without Root Complex. This is argument for using single Root
>    Complex emulated in Xen.
>
> - As far as I know (actually, Oleksandr told this to me), QEMU has no
>    mechanism for exposing virtio-pci backends without exposing PCI root
>    complex as well. Architecturally, there should be a PCI bus to which
>    virtio-pci devices are connected. Or we need to make some changes to
>    QEMU internals to be able to create virtio-pci backends that are not
>    connected to any bus. Also, added benefit that PCI Root Complex
>    emulator in QEMU handles legacy PCI interrupts for us. This is
>    argument for separate Root Complex for QEMU.
>
> As right now we have only virtio-pci backends provided by QEMU and this
> setup is already working, I propose to stick to this
> solution. Especially, taking into account that it does not require any
> changes to hypervisor code.
I am not against two hostbridge as a temporary solution as long as this 
is not a one way door decision. I am not concerned about the hypervisor 
itself, I am more concerned about the interface exposed by the toolstack 
and QEMU.
To clarify, I don't particular want to have to maintain the two 
hostbridges solution once we can use a single hostbridge. So we need to 
be able to get rid of it without impacting the interface too much.
Cheers,

--
Julien Grall