|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-devel] [RFC 01/16] docs: create Memory Bandwidth Allocation (MBA) feature document.
This patch creates MBA feature document in doc/features/. It
describes details for MBA.
Signed-off-by: Yi Sun <yi.y.sun@xxxxxxxxxxxxxxx>
---
docs/features/intel_psr_mba.pandoc | 226 +++++++++++++++++++++++++++++++++++++
1 file changed, 226 insertions(+)
create mode 100644 docs/features/intel_psr_mba.pandoc
diff --git a/docs/features/intel_psr_mba.pandoc
b/docs/features/intel_psr_mba.pandoc
new file mode 100644
index 0000000..8c04f01
--- /dev/null
+++ b/docs/features/intel_psr_mba.pandoc
@@ -0,0 +1,226 @@
+% Intel Memory Bandwidth Allocation (MBA) Feature
+% Revision 1.0
+
+\clearpage
+
+# Basics
+
+---------------- ----------------------------------------------------
+ Status: **Tech Preview**
+
+Architecture(s): Intel x86
+
+ Component(s): Hypervisor, toolstack
+
+ Hardware: MBA is supported on Skylake Server and beyond
+---------------- ----------------------------------------------------
+
+# Overview
+
+The Memory Bandwidth Allocation (MBA) feature provides indirect and approximate
+control over memory bandwidth available per-core. This feature provides OS/VMMs
+the ability to slow misbehaving apps/VMs or create advanced closed-loop control
+system via exposing control over a credit-based throttling mechanism.
+
+## Terminology
+
+* CAT Cache Allocation Technology
+* COS/CLOS Class of Service
+* MSRs Machine Specific Registers
+* PSR Intel Platform Shared Resource
+* VMM Virtual Machine Monitor
+* THRTL Throttle value or delay value
+
+# User details
+
+* Feature Enabling:
+
+ Add "psr=mba" to boot line parameter to enable MBA feature.
+
+* xl interfaces:
+
+ 1. `psr-mba-show [domain-id]`:
+
+ Show system/domain MBA information.
+
+ 2. `psr-mba-set [OPTIONS] domain-id throttling`:
+
+ Set memory bandwidth throttling for domain.
+
+ Options:
+ '-s': Specify the socket to process, otherwise all sockets are processed.
+
+ Throttling value set in register implies memory bandwidth blocked, i.e.
+ higher throttling value results in lower bandwidth. The max throttling
+ value can be got through CPUID.
+
+ The response of the throttling value could be linear mode or non-linear
+ mode.
+
+ Linear mode: the input precision is defined as 100-(MBA_MAX). For
instance,
+ if the MBA_MAX value is 90, the input precision is 10%. Values not an even
+ multiple of the precision (e.g., 12%) will be rounded down (e.g., to 10%
+ delay applied) by HW automatically.
+
+ Non-linear mode: input delay values are powers-of-two from zero to the
+ MBA_MAX value from CPUID. In this case any values not a power of two will
+ be rounded down the next nearest power of two by HW automatically.
+
+# Technical details
+
+MBA is a member of Intel PSR features, it would share some base PSR
+infrastructure in Xen.
+
+## Hardware perspective
+
+MBA provides an architectural consistent method to map cores’ to a Class
+of Service (COS). This infrastructure will be shared with the previously
+introduced CAT technologies.
+
+Furthermore, MBA also defines a new range MSRs to support specifying a
+delay value (Thrtl) per COS, with details below.
+
++----------------------------+----------------+
+| MSR (per socket) | Address |
++----------------------------+----------------+
+| IA32_L2_QOS_Ext_BW_Thrtl_0 | 0xD50 |
++----------------------------+----------------+
+| ... | ... |
++----------------------------+----------------+
+| IA32_L2_QOS_Ext_BW_Thrtl_n | 0xD50+n (n<64) |
++----------------------------+----------------+
+
+When context switch happens, the COS of VCPU is written to per-thread
+MSR `IA32_PQR_ASSOC`, and then hardware enforces bandwidth allocation
+according to the throttling value corresponding to the COS.
+
+## The relationship between MBA and CAT/CDP
+
+Generally speaking, MBA is completely independent of CAT/CDP, and any
+combination may be applied at any time, e.g. enabling MBA with CAT
+disabled.
+
+But it needs to be noticed that MBA shares COS infrastructure with CAT,
+although MBA is enumerated by different CPUID leaf from CAT (which
+indicates that the max COS of MBA may be different from CAT).
+
+## Design Overview
+
+* Core COS/Thrtl association
+
+ When enforcing Memory Bandwidth Allocation, all cores of domains have
+ the same default COS (COS0) which correspond to the same Thrtl (0).
+ The default COS is used only in hypervisor and is transparent to tool
+ stack and user.
+
+ System administrator can change PSR allocation policy at runtime by
+ tool stack. Since MBA shares COS with CAT/CDP, a COS corresponds to a
+ 2-tuple, like [CBM, Thrtl] with only-CAT enalbed, when CDP is enable,
+ the COS corresponds to a 3-tuple, like [Code_CBM, Data_CBM, Thrtl]. If
+ neither CAT nor CDP is enabled, things would be easier, one COS
+ corresponds to one Thrtl.
+
+* VCPU schedule
+
+ This part reuses CAT COS infrastructure.
+
+* Multi-sockets
+
+ Different sockets may have different MBA ability (like max COS)
+ although it is consistent on the same socket. So the capability
+ of per-socket MBA is specified.
+
+## Implementation Description
+
+* Hypervisor interfaces:
+
+ 1. Boot line param: "psr=mba" to enable the feature.
+
+ 2. SYSCTL:
+ - XEN_SYSCTL_PSR_MBA_get_info: Get system MBA information.
+
+ 3. DOMCTL:
+ - XEN_DOMCTL_PSR_MBA_OP_GET_THRTL: Get Throttling for a domain.
+ - XEN_DOMCTL_PSR_MBA_OP_SET_THRTL: Set Throttling for a domain.
+
+* xl interfaces:
+
+ 1. psr-mba-show [domain-id]
+ Show system/runtime MBA information.
+ => XEN_SYSCTL_PSR_MBA_get_info/XEN_DOMCTL_PSR_MBA_OP_GET_THRTL
+
+ 2. psr-mba-set [OPTIONS] domain-id throttling
+ Set bandwidth throttling for a domain.
+ => XEN_DOMCTL_PSR_MBA_OP_SET_THRTL
+
+* Key data structure:
+
+ 1. Feature HW info
+
+ ```
+ struct psr_mba_hw_info {
+ unsigned int thrtl_max;
+ unsigned int cos_max;
+ unsigned int linear;
+ };
+
+ - Member `thrtl_max`
+
+ `thrtl_max` is the max throttling value to be set.
+
+ - Member `cos_max`
+
+ `cos_max` is one of the hardware info of CAT.
+
+ - Member `linear`
+
+ `thrtl_max` means the response of delay value is linear or not.
+
+ As mentioned above, MBA is a member of Intel PSR features, it would
+ share some base PSR infrastructure in Xen. So, for other data structure
+ details, please refer 'intel_psr_l2_cat.pandoc'.
+
+# Limitations
+
+MBA can only work on HW which enables it (check by CPUID).
+
+# Testing
+
+We can execute these commands to verify MBA on different HWs supporting them.
+
+For example:
+ root@:~$ xl psr-hwinfo --mba
+ Memory Bandwidth Allocation (MBA):
+ Socket ID : 0
+ Linear Mode : Enabled
+ Maximum COS : 7
+ Maximum Throttling Value: 90
+ Default Throttling Value: 0
+
+ root@:~$ xl psr-mba-set 1 0xa
+
+ root@:~$ xl psr-mba-show 1
+ Socket ID : 0
+ Default THRTL : 0
+ ID NAME CBM
+ 1 ubuntu14 0xa
+
+# Areas for improvement
+
+N/A
+
+# Known issues
+
+N/A
+
+# References
+
+"INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES" [Intel®
64 and IA-32 Architectures Software Developer Manuals,
vol3](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html)
+
+# History
+
+------------------------------------------------------------------------
+Date Revision Version Notes
+---------- -------- -------- -------------------------------------------
+2017-01-10 1.0 Xen 4.9 Design document written
+---------- -------- -------- -------------------------------------------
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |