Re: [PATCH 1/2] docs/designs/launch: hyperlaunch design document

[Christopher Clark <christopher.w.clark@xxxxxxxxx>]

On Thu, Mar 25, 2021 at 9:55 AM Stefano Stabellini
<sstabellini@xxxxxxxxxx> wrote:
> On Thu, 25 Mar 2021, Roger Pau Monné wrote:
> > On Thu, Mar 25, 2021 at 10:14:31AM +0100, George Dunlap wrote:
> > >
> > > > On Mar 25, 2021, at 8:32 AM, Roger Pau Monne <roger.pau@xxxxxxxxxx> 
> > > > wrote:
> > > > On Wed, Mar 24, 2021 at 05:53:26AM -0700, Christopher Clark wrote:
> > > >> On Wed, Mar 24, 2021 at 1:01 AM Roger Pau Monné <roger.pau@xxxxxxxxxx> 
> > > >> wrote:
> > > >>> On Tue, Mar 23, 2021 at 10:39:53AM -0700, Christopher Clark wrote:
> > > >>>> On Thu, Mar 18, 2021 at 9:43 AM Roger Pau Monné 
> > > >>>> <roger.pau@xxxxxxxxxx> wrote:
> > > >>> If you offload domain creation of guests with
> > > >>> pci-passthrough devices to a control domain and/or hardware domain,
> > > >>> I'm not sure I see the difference from normal domain creation, ie:
> > > >>> it's no longer something specific to hyperlaunch, as I could achieve
> > > >>> the same by using the existing xendomains init script.
> > > >>
> > > >> So that's not what we've proposed, and hopefully not what we'll need 
> > > >> to do.
> > > >>
> > > >> Do you know if there is a need to perform work to support the
> > > >> assignment of PCI devices at the point of domain creation (ie. in
> > > >> domain_create), rather than handling it in a later step of domain
> > > >> configuration, prior to the domain being started?
> > > >
> > > > So while I think you could indeed create a domain from the hypervisor
> > > > in a paused state and attach the pci devices later from a
> > > > control/hardware domain, I don't see much benefit in doing it. If you
> > > > need to end up waiting for a control/hardware domain to attach the
> > > > devices and unpause you might as well do the whole domain creation
> > > > from such control/hardware domain.
> > >
> > > My understanding was that one of the primary advantages of domB was
> > > that you could compile and run arbitrary code in whatever language
> > > you wanted to, using already known tools.  If *all* we want to do is
> > > to assign some pre-defined specific BDFs to specific domains, then
> > > sure, we could add that capability to Xen.
> >
> > Well, it's not so easy because we require QEMU or pciback ATM on x86
> > in order to do pci passthrough to guests, so assigning BDFs to
> > specific domains from the hypervisor would need to be done using vPCI
> > (which is not yet ready for unprivileged guest usage) and only support
> > HVM kind of guests strictly.
> >
> > Doing passthrough to PV guests directly from the hypervisor is
> > impossible AFAICT without putting pciback inside of the hypervisor.
> >
> > > But suppose you wanted
> > > to have the flexibility to handle changes in hardware at boot time?
> > > “Scan through the PCI bus and assign anything that looks like a
> > > network card to domNet, and anything that looks like a USB
> > > controller to domUSB” is something you could easily do in domB, but
> > > would be way too complicated to add to Xen.
> >
> > Right, but then you might as well create the domain from domB instead
> > of doing it in the hypervisor?
> >
> > I'm not arguing about not using domB, I just don't see the benefit of
> > creating a paused domain from the hypervisor that then requires the
> > intervention of a further domain (domB) in order to finish creation.
> > Won't it be simpler to just create the domain and attach the pci
> > devices from domB?
>
> I think that the ability of creating multiple VMs from Xen is actually a
> very good one to have for a few reasons. It would align x86 with ARM, it
> would be useful in cases where PCI passthrough is not involved, and it
> is a powerful tool to have in our toolbox.
>
> I see that handling PCI passthrough at domain creation time can be
> difficult, so I think Christopher's solution is a good compromise.
>
> FYI for dom0less/ARM we have been discussing doing device assignment at
> creation time, but the idea was to provide the PCI topology in device
> tree to Xen to help with discovery.

a quick note:

- thanks for all the feedback and questions in this thread, which we
will cover in the next version of the design document

- we have made available the code developed for our prototype, first
development milestone:
https://github.com/apertussolutions/xen-hyperlaunch/tree/hyperlaunch-prototype-lcm

- we look forward to discussion on the Community Call tomorrow

Christopher