S0ix support in Xen

[Simon Gaiser <simon@xxxxxxxxxxxxxxxxxxxxxx>]

Hi,

I have been looking into using S0ix with Xen. On systems with with 11th
gen (Tiger Lake) Intel mobile CPUs or newer this is often the only
supported suspend method, thus we want to support it in Qubes OS.

Below a summary of my current understanding of what's needed (and known
unknowns). I would appreciate some feedback (what's missing, preferred
solutions, etc.).

Note this topic is much above my previous experience with Xen and x86
power management internals, so sorry if I'm missing things that are
obvious to you.

PIT timer: During some previous private discussion it was mentioned that
the PIT timer that Xen initializes for IO-APIC testing prevents S0ix
residency and therefore that part needs to be reworked. But if I'm
reading the current code correctly Xen can already use the HPET timer
instead, either with an automatic fallback if PIT is unavailable or by
forcing it via hpet=legacy-replacement=1. Looking at the rest I think
the PIT isn't used if Xen finds another clocksource. Did I miss
something?

mwait idle driver: While mwait-idle.c share a lot of code with Linux's
intel_idle.c and the imported code seems to have been updated (for
example for Alder Lake) it only supports the CPUs with hardcoded
cstates. Linux's code also has a code path to read the cstate config
from ACPI if the driver doesn't has a hard coded config for the model.
This is needed for example for Tiger Lake. For my current testing I
added the values the Linux code reads from ACPI by hand. But that's of
course no proper solution. How should this be handled in Xen (IIUC some
ACPI things are handled by Xen and some by dom0)?

The debugging code in xen/arch/x86/acpi/cpu_idle.c (and maybe other
places) need to learn about deeper package C states, for example for
Tiger Lake.

In general having the power management debugging available that you have
under plain Linux through /sys/kernel/debug/pmc_core, etc. would be
nice. Some things are as easy as allowing some dom0 to read some MSRs.
But didn't looked into it further, some functions might also be required
more complex integration (like extending xenpm's functionality).

I'm not sure yet is what else in Xen needs to learn about S0ix. Running
domains need to be halted, that can be handled by the toolstack. What
other Xen internal things need to be aware of S0ix? Like avoiding
unnecessary timer wakeups or similar. That's currently my biggest
unknown and it would be great if someone with more insight and overview
could give some hints here.

On my test system (NUC11TNHi5, Tiger Lake) I haven't seen it reaching
cstates lower PC7 with Xen (so it can of course not reach S0ix
residency). With plain Linux I got it working on that system although it
was rather fiddly and probably something is fishy on the firmware side.
After seeing it very occasionally not working under plain Linux I have
installed a newer firmware version. With that Xen currently doesn't wake
up after triggering s2idle in dom0. I'm currently looking into that and
will follow up if I have more details on that part.

Thanks, Simon

Attachment: OpenPGP_signature
Description: OpenPGP digital signature