Re: [Xen-devel] Xen HVM regression on certain Intel CPUs

[Stefan Bader <stefan.bader@xxxxxxxxxxxxx>]

On 27.03.2013 17:04, Konrad Rzeszutek Wilk wrote:
> On Wed, Mar 27, 2013 at 04:53:16PM +0100, Stefan Bader wrote:
>> On 27.03.2013 16:26, Stefan Bader wrote:
>>> Recently I ran some experiments on newer hardware and realized that when 
>>> booting
>>> any kernel newer or equal to v3.5 (Xen version 4.2.1) in 64bit mode would 
>>> fail
>>> to bring up any APs (message about CPU Stuck). I was able to normally bisect
>>> into a range of realmode changes and then manually drill down to the 
>>> following
>>> commit:
>>>
>>> commit cda846f101fb1396b6924f1d9b68ac3d42de5403
>>> Author: Jarkko Sakkinen <jarkko.sakkinen@xxxxxxxxx>
>>> Date:   Tue May 8 21:22:46 2012 +0300
>>>
>>>     x86, realmode: read cr4 and EFER from kernel for 64-bit trampoline
>>>
>>>     This patch changes 64-bit trampoline so that CR4 and
>>>     EFER are provided by the kernel instead of using fixed
>>>     values.
>>>
>>> From the Xen debugging console it was possible to gather a bit more data 
>>> which
>>> pointed to a failure very close to setting CR4 in startup_32. On this 
>>> particular
>>> hardware the saved CR4 (about to be set) was 0x1407f0.
>>>
>>> This would set two flags that somehow feel dangerous: PGE (page global 
>>> enable)
>>> and SMEP (supervisor mode execution protection). SMEP turns out to be the 
>>> main
>>> offender and the following change allows the APs to start:
>>>
>>> --- a/arch/x86/realmode/rm/trampoline_64.S
>>> +++ b/arch/x86/realmode/rm/trampoline_64.S
>>> @@ -93,7 +93,9 @@ ENTRY(startup_32)
>>>         movl    %edx, %fs
>>>         movl    %edx, %gs
>>>
>>> -       movl    pa_tr_cr4, %eax
>>> +       movl    $X86_CR4_SMEP, %eax
>>> +       notl    %eax
>>> +       andl    pa_tr_cr4, %eax
>>>         movl    %eax, %cr4              # Enable PAE mode
>>>
>>>         # Setup trampoline 4 level pagetables
>>>
>>> Now I am not completely convinced that this is really the way to go. Likely 
>>> the
>>> Xen hypervisor should not start up the guest with CR4 on the BP containing 
>>> those
>>> flags. But maybe it still makes sense to mask some dangerous ones off in the
>>> realmode code (btw, it seemed that masking the assignments in arch_setup or
>>> setup_realmode did not work).
>>>
>>> And finally I am wondering why the SMEP flag in CR4 is set anyway. My
>>> understanding would be that this should only be done if cpuid[7].ebx has 
>>> bit7
>>> set. And this does not seem to be the case at least on the one box I was 
>>> doing
>>> the bisection on.
>>
>> Seems that I was relying on the wrong source of information when checking 
>> SMEP
>> support. The cpuid command seems at fail. But /proc/cpuinfo reports it. So 
>> that
>> at least explains where that comes from... sorry for that.
> 
> OK, so if you boot Xen with smep=1 (which disables SMEP, kind of 
> counterintuive flag)
> that would work fine?

Rebooting with smep=1 as a hv argument does not fix it. But I would be careful
since I just quickly did this without checking whether Xen 4.2.1 undestands the
flag already.

Second using x86info --all on bare metal does show bits set for cpuid[7] and
/proc/cpuinfo values are consistent across BP and APs. So I am a tool for using
the wrong tool there.

So I would say the main issue to look at is why reading cr4 as a HVM guest
produces the flags on boot. Surely the hypervisor itself has set certain things
up but likely there are some epxectations about the initial setup on boot.

> 
> CC-ing the Intel folks who added this in.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel
>

Attachment: signature.asc
Description: OpenPGP digital signature

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel