Xen Summit 2025 - "Follow up Xen Toolstack new design proposal" design session notes

[Alex Brett <alex.brett@xxxxxxxxxx>]

Using device tree to describe a guest

Stefano Stabellini (SS): We're not stuck to the existing format, we can change 
it if needed. The value is in having a common interface, not the current 
implementation.

Daniel Smith (DS): Burden goes on to the toolstack to have to have the library.
Alejandro Vallejo (AV): Don't have to use the library, compilers can do it.

SS: Userspace not an issue, plenty of tools existing

Andrew Cooper (AC): At the moment we have a bunch of hypercalls, some long 
running some short running. Xen doesn't have a model where one single thread 
can run something for a minute (important for things like interrupt delivery 
etc). At boot we don't turn on this checking, but reusing hyperlaunch stuff 
when VMs are already running can't be the same code as used at boot, as you'd 
need e.g. hypercall check continuation at a minimum.

AC: At the point you have a hypercall with a DTB blob, you can't take a minute 
of time processing it.

SS/Roger Pau Monne (RPM): Long running part is populating the p2m. Also issues 
around PCI reset (spec mandated).

AC: At the moment domain create is reasonably long and can't be continued (as 
continuation needs to hang off a struct domain). Minimum amount which needs 
doing before you can start continuations.

AC: Hypercall continuations are holding a vcpu hostage.

SS: Could we have a different continuation? e.g. domain shutdown has a sequence 
of ~10 steps, and you remember which step you're in. Could have a sequence of 
steps on create that returns 'done step 1' etc.

DS: Can we look at what is the minimal stuff to get to a struct domain, and 
modify the logic so this is as short/atomic as possible so we can then get to 
continuations quickly?

AC: Currently lots gets done before you make the domain visible (e.g. max CPUs 
and event channels where Xen will expect structures to exist at the point it 
can enumerate the domain), so we might end up having to make lots of other 
modifications.

SS: Could still do the step thing, just need to make sure the step to get to a 
struct domain is quick enough to not need a continuation.

DS: Could we put a flag in struct domain that says "I'm in progress".
AC: Every single code path that enumerates domains would need modifications. Or 
maybe you make lookup return null but you'll still then need a special call for 
the genuine case.
RPM: Similar to is_dieing flag. Could we reuse that?

RPM: We should first try the simple way and see how long it takes, it may be 
fast enough.

AC: In a security issue we were forced to start allocating the p2m (limited to 
256 pages). This is allocating and zeroing memory - this is the kind of thing 
which really needs to be in a continuation (RPM: Or not done there at all).

AC: If we can split it into the minimal set and then a huge bunch of work using 
continuations that might be OK.
SS: We don't need to be religious that it has to be one call, but today we have 
a dozen and it's not clear about the ordering etc, so going down to one or two 
is still good.
AC: If you make it two there's a way to screw it up, but if the first set can 
get to a point it's safe to do continuations, you just keep on continuing until 
it comes back success or fail.

DS: This would be a new call. What if we want to do it in __init.
SS: Had a request to trim Xen down for security - started KConfig'ing things 
off, but clear we need to do something about __init.

RPM/SS: Clarified there are people who will use dom0less and want to remove the 
new call. But dom0less people will still want to have __init, so you can't take 
this code out if it's used in both places.

AC: Works if you have a binary choice if you want to keep it at runtime or not, 
but when you have 2/3 choices it doesn't.

SS: Safety will want the smallest possible amount of code, will want to KConfig 
things out. Can't KConfig out this code (as it'll break dom0less), so it needs 
to remain for init. Will still therefore need certifying anyway as stuff in 
init has to be certified, so not a massive issue.

DS: We're putting a lightweight toolstack inside the hypervisor for hyperlaunch 
- currently OK as this is only used at boot, but if you leave it there at 
runtime some people might get unhappy with this. Could you have a flag for 
adding this domain creation logic that controls it in both places?

AC: Yes, would need to think of a name that's meaningful (similar to init or 
livepatch).

AC: The argument for doing this was to get rid of the toolstack and not have 
duplicate code in the toolstack. The only way you get there eventually is to 
delete all the other hypercalls (obviously in a phased way). If we do this, the 
layout of data in DTB is an ABI.

RPM: Can we define extensions to the DTB?
DS/SS: Sort of already exists, but not defined

< Discussion around API/ABIs that I couldn't keep up with to note sensibly >