[Xen-devel] Xen Platform QoS design discussion

["Xu, Dongxiao" <dongxiao.xu@xxxxxxxxx>]

Hello maintainers,

As PQoS feature has already been reviewed for 10 rounds, and also considering 
that more QoS related features are published in new SDM, like MBM and CQE 
(refer to chapter 17.14 and 17.15 of Intel SDM Volume 3). Before sending out 
another version of patch, I'd like to make sure we are aligned on some basic 
design issues, making the final implementation simple and performance efficient.

1) The hypercall to query QoS monitoring data, should be sysctl or domctl?
Previous QoS monitoring data query hypercall is designed with sysctl, which 
returns the whole QoS data in a 2-dimenstion array format for all domains. 
However users don't like to handle such 2-dimension array themselves, but 
preferring the hypercall API to be issued per-domain, by using domain id as the 
input parameter and returns the domain related QoS data. Here I propose to use 
the domctl style hypercall to get QoS data for specific domain. This has the 
advantage of simplifying the libxl QoS APIs for user-space developers, and also 
make the QoS memory allocation in Xen much easier.

2) How much memory needs to be allocated in Xen for QoS monitoring data?
There are a lot of comments complaining the previous memory allocation for QoS 
monitoring data. But if we adopt the domctl style of hypercall proposed in 1), 
then the needed memory is much smaller since the data structure becomes 
1-dimension. We only need to allocate "nr_sockets" related size in 
initialization time to present the domain's QoS monitoring data. Furthermore, 
no additional QoS resource allocation logic is needed when CPU online/offline.

3) Copy or share for QoS monitoring data?
In previous patches, sharing mechanism is used for passing data between Xen and 
dom0 toolstack. However when MBM feature comes out, we may need to re-consider 
this solution.
CQM and MBM features belong to L3 monitoring category, and they share the same 
CPUID enumeration method for its QoS data structure, which is 64bit width. From 
data structure design point of view, all L3 monitoring category features (CQM, 
MBM and others in L3 category) should better to share the following structure:

struct socket_l3 {
    unsigned int socket_id;
    unsigned int l3_type; // CQM or MBM or others
    unsigned long qm_data_l3;
}

In hypercall, user sets the "l3_type" as an input parameter (CQM or MBM or 
others in L3 category), and Xen returns the qm_data_l3 as the monitoring data 
for that certain feature.
In this case, sharing doesn't work since data may be collapsed if there are two 
xl instances running simultaneously to separately get CQM and MBM data. 
Besides, if we use the per-domain hypercall (domctl instead of sysctl), the 
data amount is very small for every hypercall. (saying 4 sockets, the data 
amount is 4*sizeof(struct socket_l3) = 64 bytes per hypercall). Even for data 
sharing mechanism, we need to pass the page address as the parameter, which 
also requires 8 or more bytes. Therefore in my opinion, data copy is acceptable 
in PQoS case.

Any feedback is welcome!

Thanks,
Dongxiao

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel