Re: [Xen-devel] [PATCH v2 01/15] docs: create Memory Bandwidth Allocation (MBA) feature document

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Thu, Aug 24, 2017 at 09:14:35AM +0800, Yi Sun wrote:
> This patch creates MBA feature document in doc/features/. It describes
> key points to implement MBA which is described in details in Intel SDM
> "Introduction to Memory Bandwidth Allocation".
> 
> Signed-off-by: Yi Sun <yi.y.sun@xxxxxxxxxxxxxxx>
> ---
> v2:
>     - declare 'HW' in Terminology.
>       (suggested by Chao Peng)
>     - replace 'COS ID of VCPU' to 'COS ID of domain'.
>       (suggested by Chao Peng)
>     - replace 'COS register' to 'Thrtl MSR'.
>       (suggested by Chao Peng)
>     - add description for 'psr-mba-show' to state that the decimal value is
>       shown for linear mode but hexadecimal value is shown for non-linear 
> mode.
>       (suggested by Chao Peng)
>     - remove content in 'Areas for improvement'.
>       (suggested by Chao Peng)
>     - use '<>' to specify mandatory argument to a command.
>       (suggested by Wei Liu)
> v1:
>     - remove a special character to avoid the error when building pandoc.
> ---
>  docs/features/intel_psr_mba.pandoc | 256 
> +++++++++++++++++++++++++++++++++++++
>  1 file changed, 256 insertions(+)
>  create mode 100644 docs/features/intel_psr_mba.pandoc
> 
> diff --git a/docs/features/intel_psr_mba.pandoc 
> b/docs/features/intel_psr_mba.pandoc
> new file mode 100644
> index 0000000..21592e8
> --- /dev/null
> +++ b/docs/features/intel_psr_mba.pandoc
> @@ -0,0 +1,256 @@
> +% Intel Memory Bandwidth Allocation (MBA) Feature
> +% Revision 1.4
> +
> +\clearpage
> +
> +# Basics
> +
> +---------------- ----------------------------------------------------
> +         Status: **Tech Preview**
> +
> +Architecture(s): Intel x86
> +
> +   Component(s): Hypervisor, toolstack
> +
> +       Hardware: MBA is supported on Skylake Server and beyond
> +---------------- ----------------------------------------------------
> +
> +# Terminology
> +
> +* CAT         Cache Allocation Technology
> +* CBM         Capacity BitMasks
> +* CDP         Code and Data Prioritization
> +* COS/CLOS    Class of Service
> +* HW          Hardware
> +* MBA         Memory Bandwidth Allocation
> +* MSRs        Machine Specific Registers
> +* PSR         Intel Platform Shared Resource
> +* THRTL       Throttle value or delay value
> +
> +# Overview
> +
> +The Memory Bandwidth Allocation (MBA) feature provides indirect and 
> approximate
> +control over memory bandwidth available per-core. This feature provides OS/
> +hypervisor the ability to slow misbehaving apps/domains or create advanced
> +closed-loop control system via exposing control over a credit-based 
> throttling
> +mechanism.

I don't really understand what "advanced closed-loop control system
via exposing..." means. From my understand it is clear/simpler to
write it as:

"... the ability to slow misbehaving apps/domains by using a
credit-based throttling mechanism".

> +
> +# User details
> +
> +* Feature Enabling:
> +
> +  Add "psr=mba" to boot line parameter to enable MBA feature.
> +
> +* xl interfaces:
> +
> +  1. `psr-mba-show [domain-id]`:
> +
> +     Show memory bandwidth throttling for domain. For linear mode, it shows 
> the
> +     decimal value. For non-linear mode, it shows hexadecimal value.

You should first explain what are the linear and non-linear modes.

> +
> +  2. `psr-mba-set [OPTIONS] <domain-id> <throttling>`:
> +
> +     Set memory bandwidth throttling for domain.
> +
> +     Options:
> +     '-s': Specify the socket to process, otherwise all sockets are 
> processed.
> +
> +     Throttling value set in register implies memory bandwidth blocked, i.e.
> +     higher throttling value results in lower bandwidth. The max throttling
> +     value can be got through CPUID.

This is also hard to understand IMHO. I would rather write it as:

"The throttling value describes the amount of blocked bandwidth".
Although I have to admit I don't really understand this interface,
wouldn't it be easier to specify the memory bandwidth allowed
per-domain, rather the amount of bandwidth removed?

Using your approach the user has to first get the total bandwidth, and
then subtract the removed bandwidth in order to know the remaining
bandwidth for a domain.

Also, IMHO you should provide a command to print the max throttling,
remember that from Xen's PoV Dom0 is just another domain, and the
CPUID values reported to Dom0 don't need to be the same as found on
bare metal.

> +
> +     The response of the throttling value could be linear mode or non-linear
> +     mode.
> +
> +     Linear mode: the input precision is defined as 100-(MBA_MAX). For 
> instance,

What's MBA_MAX? I don't see any reference/description of it above.

> +     if the MBA_MAX value is 90, the input precision is 10%. Values not an 
> even
> +     multiple of the precision (e.g., 12%) will be rounded down (e.g., to 10%
> +     delay applied) by HW automatically.
> +
> +     Non-linear mode: input delay values are powers-of-two from zero to the
> +     MBA_MAX value from CPUID. In this case any values not a power of two 
> will
> +     be rounded down the next nearest power of two by HW automatically.
> +
> +# Technical details
> +
> +MBA is a member of Intel PSR features, it shares the base PSR infrastructure
> +in Xen.
> +
> +## Hardware perspective
> +
> +  MBA defines a range of MSRs to support specifying a delay value (Thrtl) per
> +  COS, with details below.
> +
> +  ```
> +   +----------------------------+----------------+
> +   | MSR (per socket)           |    Address     |
> +   +----------------------------+----------------+
> +   | IA32_L2_QOS_Ext_BW_Thrtl_0 |     0xD50      |
> +   +----------------------------+----------------+
> +   | ...                        |  ...           |
> +   +----------------------------+----------------+
> +   | IA32_L2_QOS_Ext_BW_Thrtl_n | 0xD50+n (n<64) |
> +   +----------------------------+----------------+
> +  ```

Are you sure you want to hardcode this n<64? Isn't there a chance this
is going to be bumped in newer hardware?

> +
> +  When context switch happens, the COS ID of domain is written to per-thread 
> MSR
> +  `IA32_PQR_ASSOC`, and then hardware enforces bandwidth allocation according
> +  to the throttling value stored in the Thrtl MSR register.
> +
> +## The relationship between MBA and CAT/CDP
> +
> +  Generally speaking, MBA is completely independent of CAT/CDP, and any
> +  combination may be applied at any time, e.g. enabling MBA with CAT
> +  disabled.
> +
> +  But it needs to be noticed that MBA shares COS infrastructure with CAT,
> +  although MBA is enumerated by different CPUID leaf from CAT (which
> +  indicates that the max COS of MBA may be different from CAT). In some
> +  cases, a domain is permitted to have a COS that is beyond one (or more)
> +  of PSR features but within the others. For instance, let's assume the max
> +  COS of MBA is 8 but the max COS of L3 CAT is 16, when a domain is assigned
> +  9 as COS, the L3 CAT CBM associated to COS 9 would be enforced, but for 
> MBA,
> +  the HW works as default value is set since COS 9 is beyond the max COS (8)
> +  of MBA.
> +
> +## Design Overview
> +
> +* Core COS/Thrtl association
> +
> +  When enforcing Memory Bandwidth Allocation, all cores of domains have
> +  the same default Thrtl MSR (COS0) which stores the same Thrtl (0). The
> +  default Thrtl MSR is used only in hypervisor and is transparent to tool 
> stack
> +  and user.
> +
> +  System administrator can change PSR allocation policy at runtime by
                        ^s
> +  tool stack. Since MBA shares COS ID with CAT/CDP, a COS ID corresponds to a
     ^ using the tool ...
> +  2-tuple, like [CBM, Thrtl] with only-CAT enalbed, when CDP is enabled,
                                              ^ enabled
> +  the COS ID corresponds to a 3-tuple, like [Code_CBM, Data_CBM, Thrtl]. If
> +  neither CAT nor CDP is enabled, things would be easier, one COS ID 
> corresponds
                                            ^ are easier, since one...
> +  to one Thrtl.
> +
> +* VCPU schedule
> +
> +  This part reuses CAT COS infrastructure.
> +
> +* Multi-sockets
> +
> +  Different sockets may have different MBA ability (like max COS)
> +  although it is consistent on the same socket. So the capability
> +  of per-socket MBA is specified.
> +
> +  This part reuses CAT COS infrastructure.
> +
> +## Implementation Description
> +
> +* Hypervisor interfaces:
> +
> +  1. Boot line param: "psr=mba" to enable the feature.
> +
> +  2. SYSCTL:
> +          - XEN_SYSCTL_PSR_MBA_get_info: Get system MBA information.
> +
> +  3. DOMCTL:
> +          - XEN_DOMCTL_PSR_MBA_OP_GET_THRTL: Get throttling for a domain.
> +          - XEN_DOMCTL_PSR_MBA_OP_SET_THRTL: Set throttling for a domain.
> +
> +* xl interfaces:
> +
> +  1. psr-mba-show [domain-id]
> +          Show system/domain runtime MBA throttling value. For linear mode,
> +          it shows the decimal value. For non-linear mode, it shows 
> hexadecimal
> +          value.
> +          => XEN_SYSCTL_PSR_MBA_get_info/XEN_DOMCTL_PSR_MBA_OP_GET_THRTL
> +
> +  2. psr-mba-set [OPTIONS] <domain-id> <throttling>
> +          Set bandwidth throttling for a domain.
> +          => XEN_DOMCTL_PSR_MBA_OP_SET_THRTL
> +
> +  3. psr-hwinfo
> +          Show PSR HW information, including L3 CAT/CDP/L2 CAT/MBA.
> +          => XEN_SYSCTL_PSR_MBA_get_info
> +
> +* Key data structure:
> +
> +  1. Feature HW info
> +
> +     ```
> +     struct {
> +         unsigned int thrtl_max;
> +         unsigned int linear;
> +     } mba_info;

Is this a domctl structure? a libxl one?

> +
> +     - Member `thrtl_max`
> +
> +       `thrtl_max` is the max throttling value to be set.
> +
> +     - Member `linear`
> +
> +       `linear` means the response of delay value is linear or not.
> +
> +     As mentioned above, MBA is a member of Intel PSR features, it would
> +     share the base PSR infrastructure in Xen. For example, the 'cos_max'
> +     is a common HW property for all features. So, for other data structure
> +     details, please refer 'intel_psr_cat_cdp.pandoc'.
> +
> +# Limitations
> +
> +MBA can only work on HW which enables it (check by CPUID).
> +
> +# Testing
> +
> +We can execute these commands to verify MBA on different HWs supporting them.
> +
> +For example:
> +    root@:~$ xl psr-hwinfo --mba
> +    Memory Bandwidth Allocation (MBA):
> +    Socket ID       : 0
> +    Linear Mode     : Enabled
> +    Maximum COS     : 7
> +    Maximum Throttling Value: 90
> +    Default Throttling Value: 0
> +
> +    root@:~$ xl psr-mba-set 1 0xa

Could you elaborate a little bit on why '0xa' is used here? IMHO The
example should provide some context.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel