Re: [Xen-devel] [PATCH v10 1/6] x86: detect and initialize Cache QoS Monitoring feature

["Jan Beulich" <JBeulich@xxxxxxxx>]

>>> On 03.04.14 at 05:54, <dongxiao.xu@xxxxxxxxx> wrote:
>>  -----Original Message-----
>> From: Jan Beulich [mailto:JBeulich@xxxxxxxx]
>> Sent: Wednesday, April 02, 2014 10:48 PM
>> To: Xu, Dongxiao
>> Cc: andrew.cooper3@xxxxxxxxxx; Ian.Campbell@xxxxxxxxxx;
>> Ian.Jackson@xxxxxxxxxxxxx; stefano.stabellini@xxxxxxxxxxxxx;
>> xen-devel@xxxxxxxxxxxxx; konrad.wilk@xxxxxxxxxx; dgdegra@xxxxxxxxxxxxx;
>> keir@xxxxxxx 
>> Subject: RE: [PATCH v10 1/6] x86: detect and initialize Cache QoS Monitoring
>> feature
>> 
>> >>> On 02.04.14 at 14:40, <dongxiao.xu@xxxxxxxxx> wrote:
>> >>  -----Original Message-----
>> >> From: Jan Beulich [mailto:JBeulich@xxxxxxxx]
>> >> Sent: Wednesday, April 02, 2014 4:37 PM
>> >> To: Xu, Dongxiao
>> >> Cc: andrew.cooper3@xxxxxxxxxx; Ian.Campbell@xxxxxxxxxx;
>> >> Ian.Jackson@xxxxxxxxxxxxx; stefano.stabellini@xxxxxxxxxxxxx;
>> >> xen-devel@xxxxxxxxxxxxx; konrad.wilk@xxxxxxxxxx; dgdegra@xxxxxxxxxxxxx;
>> >> keir@xxxxxxx 
>> >> Subject: RE: [PATCH v10 1/6] x86: detect and initialize Cache QoS 
> Monitoring
>> >> feature
>> >>
>> >> >>> On 02.04.14 at 04:17, <dongxiao.xu@xxxxxxxxx> wrote:
>> >> >> Iirc disabled CPUs (i.e. hot-pluggable ones) are still listed in MADT,
>> >> >> including their LAPIC ID (which the socket number is being calculated
>> >> >> from).
>> >> >
>> >> > I dumped one MADT table from the system, and following is one piece from
>> > it:
>> >> > If the CPU is there, then APIC ID is shown as a valid value, while CPU 
>> >> > is
>> >> > not there, then APIC ID is shown as 0xFF.
>> >> > I know we can get socket info if CPU is there (by certain calculation), 
> but
>> >> > if CPU is not there, could you help to explain how can we get the socket
>> >> > information from it?
>> >>
>> >> Now a fundamental question is whether this was on a system where
>> >> CPUs are actually hotpluggable - it's quite common for APIC IDs to be
>> >> set to invalid values when the firmware just allocates a large table
>> >> and only fills (and marks enabled) the slots for existing CPUs.
>> >>
>> >> In any event with sparse APIC ID allocations like this
>> >>
>> >> ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
>> >> ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
>> >> ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
>> >> ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
>> >> ACPI: LAPIC (acpi_id[0x10] lapic_id[0x20] enabled)
>> >> ACPI: LAPIC (acpi_id[0x12] lapic_id[0x22] enabled)
>> >> ACPI: LAPIC (acpi_id[0x14] lapic_id[0x24] enabled)
>> >> ACPI: LAPIC (acpi_id[0x16] lapic_id[0x26] enabled)
>> >>
>> >> making assumptions about the socket ID range being a subset of
>> >> the NR_CPUS one is wrong - you'll either need some data structure
>> >> other than a flat array to deal with this, or at least dimension the
>> >> thing using MAX_APICS (but you'll then realize that you end up
>> >> with an even larger array, i.e. it'll become less acceptable). And
>> >> yes, other users making invalid assumptions on the socket ID
>> >> ranges should be fixed as well (i.e. likely you would want a cleanup
>> >> patch first, and then put the your QoS stuff on top).
>> >
>> > From your above explanation, it seems that we still cannot get exact socket
>> > number from the MADT table,
>> 
>> Why not? The socket IDs are derived from the LAPIC IDs, and as long
>> as you have the full set of the latter, you can also determine the full
>> set of the former.
> 
> I dumped the MADT table from the machine that supports CPU hotplug.
> However if I remove some CPUs from the sockets, those disabled CPU's APIC ID 
> is shown as 0xFF.
> Therefore to me it is difficult to get the full set of the LAPIC IDs, so as 
> the socket IDs.

Okay, that convinces me then that we need to deal with socket IDs
being known only at hotplug time.

> If use radix tree, it is implemented as a list, where all nodes are 
> allocated by xmalloc(). It is difficult to share these radix nodes between 
> Xen and Dom0 tool stack. Besides, current libxl/libxc doesn't have such radix 
> APIs.

But you could get away with a split approach: Provide the socket ID
list via ordinary hypercall means (if this isn't already derivable from
the topology sysctl anyway), and only share the per-socket data
page(s).

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel