Re: Detecting whether dom0 is in a VM

[Jan Beulich <jbeulich@xxxxxxxx>]

On 19.04.2024 17:51, George Dunlap wrote:
> On Fri, Apr 19, 2024 at 4:29 PM George Dunlap <george.dunlap@xxxxxxxxx> wrote:
>>
>> On Fri, Jul 7, 2023 at 3:56 PM George Dunlap <george.dunlap@xxxxxxxxx> wrote:
>>>>>>> Xen's public interface offers access to the featuresets known / found /
>>>>>>> used by the hypervisor. See XEN_SYSCTL_get_cpu_featureset, accessible
>>>>>>> via xc_get_cpu_featureset().
>>>>>>>
>>>>>>
>>>>>> Are any of these exposed in dom0 via sysctl, or hypfs?
>>>>>
>>>>> sysctl - yes (as the quoted name also says). hypfs no, afaict.
>>>>>
>>>>>>  SYSCTLs are
>>>>>> unfortunately not stable interfaces, correct?  So it wouldn't be 
>>>>>> practical
>>>>>> for systemd to use them.
>>>>>
>>>>> Indeed, neither sysctl-s nor the libxc interfaces are stable.
>>>>
>>>> Thinking of it, xen-cpuid is a wrapper tool around those. They may want
>>>> to look at its output (and, if they want to use it, advise distros to
>>>> also package it), which I think we try to keep reasonably stable,
>>>> albeit without providing any guarantees.
>>>
>>>
>>> We haven't had any clear guidance yet on what the systemd team want in the 
>>> <xen in a VM, systemd in a dom0> question; I just sort of assumed they 
>>> wanted the L-1 virtualization *if possible*.  It sounds like `vm-other` 
>>> would be acceptable, particularly as a fall-back output if there's no way 
>>> to get Xen's picture of the cpuid.
>>>
>>> It looks like xen-cpuid is available on Fedora, Debian, Ubuntu, and the old 
>>> Virt SIG CentOS packages; so I'd expect most packages to follow suit.  
>>> That's a place to start.
>>>
>>> Just to take the discussion all the way to its conclusion:
>>>
>>> - Supposing xen-cpuid isn't available, is there any other way to tell if 
>>> Xen is running in a VM from dom0?
>>>
>>> - Would it make sense to expose that information somewhere, either in sysfs 
>>> or in hypfs (or both?), so that eventually even systems which may not get 
>>> the memo about packaging xen-cpuid will get support (or if the systemd guys 
>>> would rather avoid executing another process if possible)?
>>
>> Resurrecting this thread.
>>
>> To recap:
>>
>> Currently, `systemd-detect-virt` has the following  input / output table:
>>
>> 1. Xen on real hardware, domU: xen
>> 2. Xen on real hardware, dom0: vm-other
>> 3. Xen in a VM, domU: xen
>> 4. Xen in aVM, dom0: vm-other
>>
>> It's the dom0 lines (2 and 4) which are problematic; we'd ideally like
>> those to be `none` and `vm-other` (or the actual value, like `xen` or
>> `kvm`).
>>
>> It looks like ATM, /proc/xen/capabilities will contain `control_d` if
>> it's a dom0.  Simply unilaterally returning `none` if
>> /proc/xen/capabilities contains `control_d` would correct the vast
>> majority of instances (since the number of instances of Xen on real
>> hardware is presumably higher than the number running virtualized).
>>
>> Is /proc/xen/capabilities expected to stay around?  I don't see
>> anything equivalent in /dev/xen.
>>
>> Other than adding a new interface to Xen, is there any way to tell if
>> Xen is running in a VM?  If we do need to expose an interface, what
>> would be the best way to do that?
> 
> Actually, the entire code for detection is here:
> 
> https://github.com/systemd/systemd/blob/f5fefec786e35ef7606e2b14e2530878b74ca879/src/basic/virt.c
> 
> The core issue seems to be this bit:
> 
>         /* Detect from CPUID */
>         v = detect_vm_cpuid();
>         if (v < 0)
>                 return v;
>         if (v == VIRTUALIZATION_VM_OTHER)
>                 other = true;
>         else if (v != VIRTUALIZATION_NONE)
>                 goto finish;
> 
>         /* If we are in Dom0 and have not yet finished, finish with
> the result of detect_vm_cpuid */
>         if (xen_dom0 > 0)
>                 goto finish;
> 
> Basically, the code expects that dom0 will be able to read the raw
> virtualization CPUID leaves, and that the result will be
> VIRTUALIZATION_NONE on real hardware.
> 
> But in fact, the result appears to be VIRTUALIZATION_VM_OTHER -- which
> would mean that 1) the CPUID instruction for reading CPUID leaf 0x1
> succeeded, 2) the 'hypervisor' bit was set, but 3) the signature at
> the hypervisor information leaf (0x40000000) didn't match any
> hypervisor (including XenVMMXenVMM, which it's looking for).

Which in turn suggests that they (a) issue plain CPUID (not the "Xen
escaped" form of it) and (b) there's no CPUID faulting available in the
underlying hardware (or the Xen version used is too old). Since (b) is
nothing userspace can do anything about, using the "Xen escaped" form
of CPUID may be the way to go in order to actually see the hypervisor
leaves. Beyond that ...

> What do we do in regard to these for dom0?

... I'm afraid I have no good idea at the moment.

Jan