[RFC] vPMU rework proposal

[Teddy Astie <teddy.astie@xxxxxxxxxx>]

Hello,

As of today, vPMU is quite difficult to use for profiling the system, 
and has various issues. One in particular is lack of stacktrace support 
([1] exists, but is complicated partly due to ABI shortcomings, and has 
limitations).
The idea here is rather than expanding the current design would be to 
make a major rework of the feature, considering significantly the ABI 
(i.e increasing XENPMU_VER_MAJ) and simplify the design.
A important use-case of vPMU is to profile the entire system (including 
Xen itself), which is only possible if the PMU covers the entire system 
(and not just Dom0), and if the Dom0 can use the PMU consistently. The 
idea here is to have a mode where the PMU registers are "global" 
(system-wide) and requires pinned Dom0 vCPUs.
In the current design, the PMU registers are stashedin the pmu page, and 
synced on PMI. The issue is that the for several  of these registers, 
their count is uarch-dependant (and even SKU dependent), which makes the 
layout of the shared page complex.
A proposed alternative would be to not have PMU registers in the page 
and rely on either a new dedicated hypercall (PV/PVH Dom0) or directly 
use RDMSR/WRMSR (PVH Dom0) to interact with them instead.
With that new layout, the hypervisor stacktrace can be placed right 
below the other fields of the shared PMU page, instead of at the bottom 
of the page, which makes things much simpler.
Another idea I had would be to expose rdpmc to PV Dom0 (CR4.PCE), but 
I'm not sure how useful it's going to be in practice.
In this new design, at least in the current proposal, either only PV/PVH 
Dom0 has access to vPMU, or select PVH/HVM have access to their own 
vPMU. There is no longer support for vPMU for PV DomU anymore.
Teddy

[1] 
https://lore.kernel.org/xen-devel/cover.1753372928.git.edwin.torok@xxxxxxxxx/T/#m9507ad8840515503f736823d0d699fb1b8ffa7a1
---

# Proposal (“vPMU v2”)

There are roughly 2 (exclusive) modes for vPMU :

- Profile the system as a whole, driven by Dom0
- Virtualize PMU in HVM guests (non-security supported)
  - Use hardware facilities (AMD PMC Virtualization / Intel "load 
`IA32_PERF_GLOBAL_CTRL`" VMX control)
These 2 use-cases are exclusive and needs to be chosen at boot time i.e :

- `vpmu=off` / `no-vpmu` : vPMU disabled (default)
- `vpmu=global` : Native PMU driven by Dom0, requires `dom0_vcpus_pin`
  - → let Dom0 profile the system
  - effectively similar to `XENPMU_MODE_ALL`
- `vpmu=hvm` : Possibility to virtualize vPMU per PVH/HVM guest (with 
`XEN_DOMCTL_CDF_vpmu`)
  - → make profilers work in guests

## Global PMU for Dom0 (vpmu=global)

Guest (Dom0) observe this capability through (to be introduced) 
`XENFEAT_vpmu`.
Additional information (stacktrace, domid, guest registers, ...) could 
be exposed through a optional (simplified) shared page per CPU.
### PV

PV domains will interact (e.g write) to PMU registers through a new 
dedicated hypercall (`XENPMU_{read,write}_regs`).
These PV domains will get `CR4[PCE]=1` (at least, for "kernel-mode" and 
are expected to read performance counters through `RDPMC` instruction 
(which is more efficient than through hypercalls).
PMI are forwarded through `VIRQ_XENPMU`.

### PVH

PVH Dom0 will observe "all" PMU registers non-intercepted and is able to 
control it through usual means (`wrmsr/rdmsr/rdpmc`), along with 
`XENPMU_{read,write}_regs`.
Xen will ensure that the PMI is forwarded through the architectural way 
(and also with `VIRQ_XENPMU` ?).
## PMU for HVM guests (vpmu=hvm)

TODO

## Notes

Intel PEBS/DS_AREA/BTS is out of scope and will not be exposed at least 
for global PMU for the time being.

Attachment: OpenPGP_signature.asc
Description: OpenPGP digital signature