Re: [Xen-devel] [ARM] Native application design and discussion (I hope)

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Thu, 6 Apr 2017, Volodymyr Babchuk wrote:
> Hello all,
> 
> I want to discuss EL0 (native) applications for XEN. This will be relatively
> long e-mail with requirements, proposed design and my PoC results.
> 
> So, why we want XEN native applications in the first place? I see the 
> following
> reasons:
> 
> 1. Isolation. I see XEN as a sort of micro-kernel, so there are no place for
> device drivers, emulators, specific SMC handlers, hypervisor extension, etc..
> 
> 2. Modularity. Just look at Linux kernel. Obviously, for different devices we
> can load different drivers.
> 
> 3. Performance. Native application should be faster than stub domain, or there
> will be no sense in it.
> 
> 4. Ease of use. I want to make call to EL0 app as easy as possible.
> Ideally - as a function call.
> 
> Actually, no one wants extra code in hypervisor, so reasons (1) and (2) are 
> most
> important. I know that there was tries to do such thing in x86 but with
> different approach. I want describe my idea for arm64.
> 
> Native application is an another domain type. It has own vCPU (only one at 
> this
> moment) Native app is loaded as any other kernel, using ELF loader.
> It looks like another stub-domain such as MiniOS, but there are two big
> differences:

Could you describe the reason why you are suggesting it? Unless strictly
necessary, I wouldn't go down the vcpu route, because as soon as we
bring a vcpu into the picture, we have a number of problems, including
scheduling, affinity, etc. It is also user-visible (xl vcpu-list) which
I don't think it should be.

I understand that one of the goals is "Modularity", which makes us think
of an ELF loader, such as the one for a new domain. I agree that
modularity is important, but I would solve it as a second step. In first
instance, I would limit the scope to run some code under
/xen/arch/arm/apps or, better, /apps (for example) in a lower privilege
mode. After that is done and working, I am sure we can find a way to
dynamically load more apps at run time.

 
> 1. MiniOS has event loop that serves requests from hypervisor. Native
> application does not has such loop. It has one entry point where you jump 
> every
> time when you need something from it.
> 
> 2. Native application runs in EL0 mode, so it does not have access to MMU,
> it can't handle vIQRs, exceptions and so on. XEN does all this for it.
>
> You can find example native application at [1]. I used exactly this one to
> benchmark my implementation. Mostly it is inspired by approach used in TEE.
> Actually, I took some code directly from OP-TEE Trusted Application library.
> In app_entry.c you can find entry point - __app_entry(). It takes function
> number and some parameters that will be passed to a function. I probably going
> to change ABI a bit, but basic idea will be the same.
> 
> Function number will be something like APP_INIT, APP_HANDLE_SMC
> or APP_HANDLE_MMIO... I think you got the idea. I also implemented two 
> syscalls
> (via old plain SVC instruction). app_log() writes to XEN log, app_return() 
> exits
> from application back to hypervisor. We will need other syscalls like
> app_call_smc(), app_map_guest_page(), app_map_io(), etc.
> 
> Now, back to XEN. Classic way to handle something with stubdomain is to
> write request to a ring buffer, fire an event through event channel, that will
> trigger vIRQ in stubdomain and stubdomain's vCPU will be scheduled to handle
> a request. Problem it that you can't control scheduler, so you don't know
> when your request will be really handled, which in not fine in some
> embedded use cases.
> 
> There is how I see handling requests with native application:
> 
> 0. Hypervisor pauses requester vCPU
> 1. Hypervisor either passes parameters via registers or writes request to a
> shared page/ring buffer.
> 2. Then in sets PC of native app vCPU to entry point and initializes r0-r7
> with function code and other parameters.
> 3. Hypervisor switches context to native app vCPU
> 4. When native app finishes request handling it calls special syscall 
> app_exit()
> 5. Hypervisor analyses return code, updates requester vCPU state (if needed),
> switches back to that vCPU, unpauses it.
> 
> Most of that was done at [2]. Most interesting part is in arch/arm/domain.c
> There are functions call_el0_app() and return_from_el0_app() that do most
> of the work. Also I have added syscalls handlers (in the same way,
> as hypercalls are handled). You can find them at xen/arch/arm/syscall.c

This workflow is actually kind of OK. I would not use the term "vcpu"
for anything related to an el0 app. Maybe we need to introduce a new
concept, such as "app_context" for example. But I would not want to
confuse "vcpu" which is the runtime environment exposed to guests, with
the el0 Xen context.

A vcpu is expected to be running simultenously with other vcpus of the
same domain or different domains. The scheduler is expected to choose
when it is supposed to be running. On the other end, an el0 app runs to
handle/emulate a single request from a guest vcpu, which will be paused
until the el0 app finishes. After that, the guest vcpu will resume.


> At this moment entry point is hardcoded and you need to update it every time
> you rebuild native application. Also there are no actual parameters passed.
> Also, whole code is a piece of gosa, because it was first time I hacked XEN.

:-)
I would start by introducing a proper way to pass parameters and return
values.


> I don't want to repeat benchmark results, because they already was posted in 
> ML.
> You can find them at [3].
> 
> I understand that I have missed many things:
> 
> 1. How to ship and load native app, because some of them will be needed even
> before dom0 is created.

I envision something like Linux's insmod, but I suggest postponing this
problem. At the moment, it would be fine to assume that all apps need to
be built statically and cannot be loaded at runtime.


> 2. How to distinguish multiple native apps

Each apps need to specify a range of MMIO/SMC handlers. Xen will invoke
the right one.


> 3. Concurrency in native apps

This is an interesting problem: what do we do if two guest vcpus make
simultenous requests that need to be handled by the same app?
Technically, we could run the same app twice on two different pcpus
simultenously. But then, the apps would need to be able to cope with
concurrency (spin_locks, etc.) From Xen point of view, it should be OK
though.


> 4. How to restart misbehaved apps.

A related question is the following: do we expect to allocate each app
once at boot or once per guest? Apps need to have some per-domain
context, but it could be passed from Xen to the app on a shared page,
possibly reducing the need for allocating the same app once per guest? 


> But at this moment I want to discuss basic approach. If there are will be no
> objections against basic concept, then we can develop details.
> 
> [1] https://github.com/lorc/xen_app_stub - native app
> [2] https://github.com/lorc/xen/tree/el0_app - my branch with PoC
> [3] http://marc.info/?l=xen-devel&m=149088856116797&w=2 - benchmark results

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel