[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH 1/2] x86/entry: Introduce POP_GPRS


  • To: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
  • From: Jan Beulich <jbeulich@xxxxxxxx>
  • Date: Thu, 14 Mar 2024 09:17:46 +0100
  • Autocrypt: addr=jbeulich@xxxxxxxx; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL
  • Cc: Roger Pau Monné <roger.pau@xxxxxxxxxx>, Wei Liu <wl@xxxxxxx>, Xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>
  • Delivery-date: Thu, 14 Mar 2024 08:18:06 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On 13.03.2024 15:26, Andrew Cooper wrote:
> The macro named RESTORE_ALL has several problems.  It adjusts the stack
> pointer despite this not being clear to the caller.  It also goes against
> recommendations in the optimisation guides because of trying to do too many
> things at once.  (i.e. there's a reason why compilers don't emit code looking
> like this.)

Not anymore; I'm sure they used to over a certain period of time, which is
why 4d246723a85a ("x86: use MOV instead of PUSH/POP when saving/restoring
register state") was created in the first place (and which you now say was
a mistake, or at least has become a mistake in the over 10 years since then).

> Introduce a new POP_GPRS macro which only POPs GPRs.  Use it for the HVM paths
> which are already using POPs.
> 
> Also use it for restore_all_{xen,guest}().  This saves most of a cacheline
> worth of code from two fastpaths:
> 
>   add/remove: 0/0 grow/shrink: 0/2 up/down: 0/-99 (-99)
>   Function                                     old     new   delta
>   restore_all_guest                            378     330     -48
>   restore_all_xen                              165     114     -51
> 
> but it also avoids having an explicit modification to the stack pointer
> between %rsp-relative accesses, which avoids stalls in the stack-engine
> optimisations in some microarchitectures.

Is there such a rule? All I was able to find (and even that only with
quite a bit of effort, because the section it's in wouldn't have had
me think of a stack pointer rule being there) is

"Assembly/Compiler Coding Rule 22. (M impact, M generality) Avoid
 putting explicit references to ESP in a sequence of stack operations
 (POP, PUSH, CALL, RET)."

I actually wonder whether %rsp really is special in the way you
indicate - other registers, when used for memory accesses and being
updated ought to have a similar stall issue?

> --- a/xen/arch/x86/hvm/svm/entry.S
> +++ b/xen/arch/x86/hvm/svm/entry.S
> @@ -74,22 +74,9 @@ __UNLIKELY_END(nsvm_hap)
>          ALTERNATIVE "", svm_vmentry_spec_ctrl, X86_FEATURE_SC_MSR_HVM
>          ALTERNATIVE "", DO_SPEC_CTRL_DIV, X86_FEATURE_SC_DIV
>  
> -        pop  %r15
> -        pop  %r14
> -        pop  %r13
> -        pop  %r12
> -        pop  %rbp
>          mov  VCPU_svm_vmcb_pa(%rbx),%rax
> -        pop  %rbx
> -        pop  %r11
> -        pop  %r10
> -        pop  %r9
> -        pop  %r8
> -        pop  %rcx /* Skip %rax: restored by VMRUN. */
> -        pop  %rcx
> -        pop  %rdx
> -        pop  %rsi
> -        pop  %rdi
> +
> +        POP_GPRS rax=%rcx /* Skip %rax.  It's restored by VMRUN. */

In light of you having asked my to try and decouple ABI and internal stack
frame layout, I'm wary of encoding a dependency on the ordering of registers
in the frame at a use site like this one. Imo the argument ought to merely
indicate "skip %rax", with the macro taking care of how that skipping is
actually carried out.

> @@ -696,20 +697,19 @@ UNLIKELY_END(exit_cr3)
>          /* WARNING! `ret`, `call *`, `jmp *` not safe beyond this point. */
>          SPEC_CTRL_EXIT_TO_XEN /* Req: %r12=ist_exit %r14=end %rsp=regs, 
> Clob: abcd */
>  
> -        RESTORE_ALL adj=8
> +        POP_GPRS
>  
>          /*
>           * When the CPU pushed this exception frame, it zero-extended eflags.
>           * For an IST exit, SPEC_CTRL_EXIT_TO_XEN stashed shadow copies of
>           * spec_ctrl_flags and ver_sel above eflags, as we can't use any 
> GPRs,
>           * and we're at a random place on the stack, not in a CPUFINFO block.
> -         *
> -         * Account for ev/ec having already been popped off the stack.
>           */
>          SPEC_CTRL_COND_VERW \
> -            scf=STK_REL(EFRAME_shadow_scf, EFRAME_rip), \
> -            sel=STK_REL(EFRAME_shadow_sel, EFRAME_rip)
> +            scf=STK_REL(EFRAME_shadow_scf, EFRAME_error_code), \
> +            sel=STK_REL(EFRAME_shadow_sel, EFRAME_error_code)
>  
> +        add   $8, %rsp
>          iretq

How is this ADD different from the RESTORE_ALL one, in particular in light
of the ORM rule quoted above (which surely extends to IRET as well)? It
ought to be possible to avoid, by having POP_GPRS (optionally) move the
%r15 value into the error code slot first thing (i.e. before %rsp starts
being updated), and then having "pop %r15" last.

Jan



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.