[Xen-devel] Xen HVM regression on certain Intel CPUs

[Stefan Bader <stefan.bader@xxxxxxxxxxxxx>]

Recently I ran some experiments on newer hardware and realized that when booting
any kernel newer or equal to v3.5 (Xen version 4.2.1) in 64bit mode would fail
to bring up any APs (message about CPU Stuck). I was able to normally bisect
into a range of realmode changes and then manually drill down to the following
commit:

commit cda846f101fb1396b6924f1d9b68ac3d42de5403
Author: Jarkko Sakkinen <jarkko.sakkinen@xxxxxxxxx>
Date:   Tue May 8 21:22:46 2012 +0300

    x86, realmode: read cr4 and EFER from kernel for 64-bit trampoline

    This patch changes 64-bit trampoline so that CR4 and
    EFER are provided by the kernel instead of using fixed
    values.

From the Xen debugging console it was possible to gather a bit more data which
pointed to a failure very close to setting CR4 in startup_32. On this particular
hardware the saved CR4 (about to be set) was 0x1407f0.

This would set two flags that somehow feel dangerous: PGE (page global enable)
and SMEP (supervisor mode execution protection). SMEP turns out to be the main
offender and the following change allows the APs to start:

--- a/arch/x86/realmode/rm/trampoline_64.S
+++ b/arch/x86/realmode/rm/trampoline_64.S
@@ -93,7 +93,9 @@ ENTRY(startup_32)
        movl    %edx, %fs
        movl    %edx, %gs

-       movl    pa_tr_cr4, %eax
+       movl    $X86_CR4_SMEP, %eax
+       notl    %eax
+       andl    pa_tr_cr4, %eax
        movl    %eax, %cr4              # Enable PAE mode

        # Setup trampoline 4 level pagetables

Now I am not completely convinced that this is really the way to go. Likely the
Xen hypervisor should not start up the guest with CR4 on the BP containing those
flags. But maybe it still makes sense to mask some dangerous ones off in the
realmode code (btw, it seemed that masking the assignments in arch_setup or
setup_realmode did not work).

And finally I am wondering why the SMEP flag in CR4 is set anyway. My
understanding would be that this should only be done if cpuid[7].ebx has bit7
set. And this does not seem to be the case at least on the one box I was doing
the bisection on.

-Stefan

Attachment: signature.asc
Description: OpenPGP digital signature

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel