Re: [Xen-devel] [ARM] Native application design and discussion (I hope)

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Mon, 24 Apr 2017, Andrii Anisov wrote:
> On 21.04.17 23:58, Stefano Stabellini wrote:
> > Hello Andrii,
> > 
> > could you please use plain text (not HTML) in your emails?
> My bad. Will be checking delivery format settings carefully.
> 
> > On Fri, 21 Apr 2017, Andrii Anisov wrote:
> > > We will also need another type of application: one which is periodically
> > > called by XEN itself, not actually servicing any domain request. This is
> > > needed for a
> > > coprocessor sharing framework scheduler implementation.
> > EL0 apps can be a powerful new tool for us to use, but they are not the
> > solution to everything. This is where I would draw the line: if the
> > workload needs to be scheduled periodically, then it is not a good fit
> > for an EL0 app.
> From my last conversation with Volodymyr I've got a feeling that notions "EL0"
> and "XEN native application" must be pretty orthogonal.
> In [1] Volodymyr got no performance gain from changing domain's exception
> level from EL1 to EL0.
> Only when Volodymyr stripped the domain's context  abstraction (i.e. dropped
> GIC context store/restore) some noticeable results were reached.
> So I treat his results as a "light domain" experiment.
> For me it is interesting how configurable and light the "domain" abstraction
> could be and when it starts to be treated as a "native application"?
> 
> > In that case, stubdoms or regular driver domains are a
> > better choice.
> Sorry for my ignorance, but is there any difference between regular domains
> and stub domains from hypervisor point of view?
> 
> [1]http://marc.info/?l=xen-devel&m=149088856116797&w=2  - benchmark results

Let me add more context and information.


Stubdomains (stubdoms in short) are small domains, each running a single
application. Typically they run unikernels rather than a full fledged
operating system. A classic example is QEMU stubdoms on x86: one QEMU
stubdoms is started for each regular guest domain. Each QEMU stubdom
instance provides emulation for one guest - it runs one instance of
QEMU.

From Xen point of view, they are regular domain, except that they are
privilege in regards to one particular guest only (they can map a page
of the guest they are servicing, but they cannot map any random page in
memory). If you do "xl list" you would see stubdoms in the output.

The advantages of using stubdoms are:
- they already exist
- their security properties are well known
- you could run almost anything, including Linux or a full distro, as a
  stubdom
- they could access hardware resources if you assign them to stubdoms
- there could be one for all guests (like DomD), or many
- they can run in parallel to the guest vcpus

The disadvantages are:
- they are scheduled independently: the Xen scheduler has no special
  code to handle stubdoms. Time spent in a stubdom is not accounted to
  the guest that triggered it.
- they potentially run on any pcpu: because they are scheduled
  independently, they could be run anywhere
- slower to context switch
- we don't have a good (any?) infrastructure to build stubdoms today
- they show up on "xl list", most users find it confusing


In this thread, we are discussing whether it makes sense to introduce a
*new* model, one that has different properties, therefore different
advantages and disadvantages. Let's call it EL0 apps.

An EL0 app is an application that runs on top of the Xen hypervisor
directly in EL0 mode. It is not a domain, and it doesn't show up on "xl
list". It runs on the same timeslot of the guest vcpu it is servicing.
It is scheduled deterministically: right after a guest vcpu traps into
the hypervisor. The build is simple. Writing an EL0 app should be
(almost) as simple as writing a regular emulator for Xen. An EL0
app is not scheduled, but it could register a timer with Xen. The
interface between EL0 apps and Xen is small: the number of
functionalities exposed are very limited. They don't access hardware.

The advantages of using EL0 apps are:
- scheduled deterministically
- faster context switch
- lower and deterministic latency
- EL0 apps execution time is accounted appropriately to the guest that
  they are servicing
- they run on the same pcpu as the guest vcpu
- writing and building EL0 apps should be easier

The disadvantages are:
- support for EL0 apps still needs to be written
- they only expose limited functionalities (map guest pages, send
  interrupts to guest, register timer)
- they don't have access to hardware
- one for each guest (not one for all guests)


As you can imagine, depending on what you need to write, one model could
be a better fit than the other. Although stubdoms on x86 are mostly used
to run QEMU, to provide emulation to guests, actually, emulators would
be a better fit as EL0 apps.

Something more complex that requires interaction with the hardware is
likely better as a standalone stubdom.


> If we are speaking about shared coprocessors framework, we need here several
> things:
>  - MMIO access emulation

This could be run as EL0 app.


>  - periodic actions (scheduling) which at least will include IOMMU
> reconfiguration and some actions with coprocessor hardware itself in runtime
>  - coprocessor interrupts handling and redistribution to target domains

These would be better as stubdoms. Or, if they are simple enough, in
the hypervisor directly.

Assuming that MMIO access emulation is done in an EL0 app, how many
types of calls does it need to make to these two components? If the
answer is none, or *very* few, then it is OK to run MMIO access
emulation as an EL0 app. These two components could even be in Xen, if
they are simple enough.

If these two pieces are run as a stubdom, we could do everything in a
stubdom (including MMIO access emulation), or we could still run MMIO
access emulation in an EL0 app, and introduce a ring buffer connecting
EL0 apps with the stubdom. EL0 apps could send requests to it that way.

If these two pieces are in Xen, we would need a very strong and
well-thought-out set of checks to protect Xen from bad EL0 calls. This
is why we are saying that if the number of EL0 calls is too high, then
it might be not be feasible to use an EL0 app for your use case.

As far as I can tell, without looking at the existing code, the options
are:
1) everything in a stubdom
2) MMIO access emulation as EL0 app, the rest in a stubdom
3) MMIO access emulation as EL0 app, the rest in Xen

The right approach depends on the type of interactions between MMIO
access emulation and the rest of the shared coprocessors framework. Does
this make sense?


> From one and we would like to have that stuff encapsulated in some container
> (application), somehow separated from XEN. From other hand minimal overhead is
> desired.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel