Design Session Notes - "How best to upstream Bareflank's implementation of the Xen VMM into the Xen Project"

[Brendan Kerrigan <brendank310@xxxxxxxxx>]

# Design Session Notes - "How best to upstream Bareflank's implementation of the Xen VMM into the Xen Project"
## Design Session Description
Assured Information Security, Inc. has been working on a new implementation of the Xen VMM (i.e., just the root component of the Xen hypervisor) using Bareflank/Boxy (https://github.com/Bareflank/hypervisor). The goals of this VMM include the ability to reload the hypervisor without having to reboot, support for a Windows Dom0 (or any Dom0 really), removal of the Xen scheduling and power management code and instead using the scheduler and power management logic built into the Dom0 kernel, and removal of PV support in favor of a pure PVH/HVM implementation. Although there is still a lot of work to do, we can demonstrate this capability today. The goal of this design session is to discuss the design of our new approach, ways in which we can improve it, and ultimately how best to upstream our work into the Xen Project.

## Current Status of Xen compatibility in Bareflank
Bareflank has compatibility layer to work with the Xen PVH hypercalls, with the goal of re-using existing Xen VMs, improved performance, less complex scheduling and power management, and the ability to run non-traditional dom0s, such as Windows.
* Prototype is up and running:
  * Removal of scheduling/power management code
  * Windows dom0
  * Hotloading the hypervisor (optional late launch, is desirable by gwd)
  * Doesn't share any code directly with Xen at the moment, was referenced throughout the implementation
* No legacy PV mode, PVH only (currently) though HVM is planned down the road
* APIC not managed by the hypervisor, but by dom0
* Mostly MSIs are passed directly to dom0
* Minimal interference with power management, lets the operating system deal with it
* Removal of possible schedule aliasing
* libxl toolstack runs outside of dom0

## Discussion of upstreamability (desirable by andyhhp)
* How to do it organizationally:
  * Subproject (feedback indicated this was undesirable, there would be too much overhead)
  * Directly bring capabilities into the mainline (feedback indicated this was more desirable)
* Potential technical requirements:
  * Add scheduling API to allow the use of a host OS scheduler rather than the built in Xen scheduler (may be difficult)

## Proposed Actions:
* KConfig options to strip out undesirable code portions
* Add ability to do the rootkit like loading of Xen (like hxen/bareflank support)
* Addition of more hypercalls to support dom0 (or more generally non-hypervisor context) scheduling
* Stay engaged with xen-devel to align changes with potential ABI changes that are coming down the road to support updating libxl and the hypervisor in stages and supporting encrypted memory schemes that are incorporated in newer architectures