Re: Xen Summit 2025 - Design Discussion Notes - Xen ABI

[Alex Brett <alex.brett@xxxxxxxxxx>]

I also took notes during this session - as usual apologies for any misquotes 
etc:

Christopher Clark (CC): Preparing a collection of docs to understand the 
problem space and what we want to achieve ahead of doing the work
- List of API/ABIs
- List of known limitations / issues / concerns with them, indexed back to the 
API/ABIs

Andrew Cooper (AC): Previously tried addressing some of the issues, but 
addressing only part of it results in a lot of the feedback being "what about 
this other thing?" etc. Designs presented so far fix part/some of the problems, 
but not all. Trying this time to identify all the problems before identifying a 
design to fix them.

First enumerate the APIs and ABIs that currently exist. Set of problems not 
apparent when you just think about API/ABIs, e.g. people think it's "just 
hypercalls", but it's not, there's things like enumeration information. Long 
history of bugs due to Xen originally being a monorepo and having forks of 
kernel/qemu etc, so many ABIs not properly defined/enumerated. Also e.g. 
initial state of vCPUs is an ABI. ~46 hypercalls, but around half are specific 
to PV guests. Hypercalls are the way they are as Xen started with x86 PV 
guests, HVM guests were created by doing minimal adjustments to PV and thus 
resulted in lots of legacy poor choices (e.g. having to walk the guest page 
tables).

Other examples:
- evtchn_send passes a 32-bit pointer to the event channel to use, rather than 
just the event channel id.
- hypercalls have a shift by 12 assuming 4k pages, causes problems in ARM
- Changing the version of Xen breaks userspace (intentional choice early on, 
but causes user pain) - unstable ABIs are killing Xen
- Security patches nominally/officially require rebuilding userspace (including 
qemu)

This is relevant to work like host UEFI secure boot (introduces new privilege 
boundary - admin with root in dom0 can't violate the MS defined boundary of not 
reading/writing arbitrary memory, hypercalls work currently with 
/dev/xen/privcmd which entirely violates this).

All problems compound, want to look at all of them before figuring out 
solutions.

Bertrand Marquis (BM): Need to look at problem discussed yesterday - how we 
create/configure a guest (two ways currently, dom0less and xl), duplicating 
code and configuration format etc. If modifying ABI between dom0 and Xen, can 
we have a coherent format we can use in multiple places to solve problems like 
this and prevent duplication / reduce required hypercalls to create a guest.

Alexander Merrit (AM): Regular applications don't make syscalls directly they 
use libraries, is that an option?
AC: Xen has libraries, but they currently need to be recompiled every time Xen 
changes
Jan Beulich (JB): This could be solved by having a user library vs the library 
that actually calls into the hypervisor
AC: Libraries currently all C, would like libraries for other languages that 
are stable. Need to consider all these possibilities when designing a solution.

JB: Concern about taking a global approach - will changing everything in one go 
mean we never end?
AC: Idea is to come up with one global design. Parties have specific interests 
(e.g. XS around secure boot, Vates for SEV) that will lead to implementation, 
but we should agree the approach up front rather than having people pulling in 
different directions.

JB: Could we deal with e.g. the hypercall problem up front rather than having 
to redesign everything else at the same time?
AC: Don't need e.g. function prototypes, but need the broad strokes agreed to 
ensure things don't diverge later on.

AC: Important to ensure backwards compatibility (can't break HVM guests)

AM: Is the problem unique to us, or are there historical references we can copy?
AC: No one problem we've found is unique, but there's enough overlap between 
problems in different areas that we can't take something off the shelf
BM: Potential problems with lots of compatibility code etc - do we still need 
it? (Room: Yes)

AC/JB/BM: discussion around whether to use physical or virtual addresses from 
PV guests. Currently made HVM consistent with PV, should it be the other way 
round?

Rich Persaud (RP): Unlikely this task can ever be completed, but any work done 
will be very useful to people after this task has failed. Xen already forked 
once (Bromium), other hypervisors focusing on specfic verticals etc. Xen is 
last entity standing trying to pull all stakeholders together, unclear how long 
this will last, but the products which will last the longest are certified 
items, and whatever snapshot is taken at that point will thus be the things 
which last the longest.

Andrei Smenov (AS): Can Xen and guest negotiate which version to use?
AC/JB: Not really, can be multiple instances within a guest (bootloader, 
kernel, kexec kernels etc). Another example OVMF coming up to do boot services 
(blkback) load a file then handing over to the kernel etc (can leave shared 
info page in a bad state)

AC: Shared info page another problem by itself. Layout done with unsigned longs 
which change size between 32 and 64 bit code, so layout of shared info page 
changes. Different vCPUs can be in 32 or 64 bit mode so we cache the mode of 
the vCPU at the point it makes one of two hypercalls, so final vCPU to crash / 
go through kexec hypercall will change the format of the page. Something we 
need to address. Whatever we provide must have in mind aspects like this.