Re: [Xen-devel] [PATCH v10 1/6] x86: detect and initialize Cache QoS Monitoring feature

["Xu, Dongxiao" <dongxiao.xu@xxxxxxxxx>]

> -----Original Message-----
> From: xen-devel-bounces@xxxxxxxxxxxxx
> [mailto:xen-devel-bounces@xxxxxxxxxxxxx] On Behalf Of Jan Beulich
> Sent: Thursday, April 03, 2014 4:19 PM
> To: Xu, Dongxiao
> Cc: keir@xxxxxxx; Ian.Campbell@xxxxxxxxxx; andrew.cooper3@xxxxxxxxxx;
> stefano.stabellini@xxxxxxxxxxxxx; Ian.Jackson@xxxxxxxxxxxxx;
> xen-devel@xxxxxxxxxxxxx; dgdegra@xxxxxxxxxxxxx
> Subject: Re: [Xen-devel] [PATCH v10 1/6] x86: detect and initialize Cache QoS
> Monitoring feature
> 
> >>> On 03.04.14 at 05:54, <dongxiao.xu@xxxxxxxxx> wrote:
> >>  -----Original Message-----
> >> From: Jan Beulich [mailto:JBeulich@xxxxxxxx]
> >> Sent: Wednesday, April 02, 2014 10:48 PM
> >> To: Xu, Dongxiao
> >> Cc: andrew.cooper3@xxxxxxxxxx; Ian.Campbell@xxxxxxxxxx;
> >> Ian.Jackson@xxxxxxxxxxxxx; stefano.stabellini@xxxxxxxxxxxxx;
> >> xen-devel@xxxxxxxxxxxxx; konrad.wilk@xxxxxxxxxx; dgdegra@xxxxxxxxxxxxx;
> >> keir@xxxxxxx
> >> Subject: RE: [PATCH v10 1/6] x86: detect and initialize Cache QoS 
> >> Monitoring
> >> feature
> >>
> >> >>> On 02.04.14 at 14:40, <dongxiao.xu@xxxxxxxxx> wrote:
> >> >>  -----Original Message-----
> >> >> From: Jan Beulich [mailto:JBeulich@xxxxxxxx]
> >> >> Sent: Wednesday, April 02, 2014 4:37 PM
> >> >> To: Xu, Dongxiao
> >> >> Cc: andrew.cooper3@xxxxxxxxxx; Ian.Campbell@xxxxxxxxxx;
> >> >> Ian.Jackson@xxxxxxxxxxxxx; stefano.stabellini@xxxxxxxxxxxxx;
> >> >> xen-devel@xxxxxxxxxxxxx; konrad.wilk@xxxxxxxxxx;
> dgdegra@xxxxxxxxxxxxx;
> >> >> keir@xxxxxxx
> >> >> Subject: RE: [PATCH v10 1/6] x86: detect and initialize Cache QoS
> > Monitoring
> >> >> feature
> >> >>
> >> >> >>> On 02.04.14 at 04:17, <dongxiao.xu@xxxxxxxxx> wrote:
> >> >> >> Iirc disabled CPUs (i.e. hot-pluggable ones) are still listed in 
> >> >> >> MADT,
> >> >> >> including their LAPIC ID (which the socket number is being calculated
> >> >> >> from).
> >> >> >
> >> >> > I dumped one MADT table from the system, and following is one piece
> from
> >> > it:
> >> >> > If the CPU is there, then APIC ID is shown as a valid value, while 
> >> >> > CPU is
> >> >> > not there, then APIC ID is shown as 0xFF.
> >> >> > I know we can get socket info if CPU is there (by certain 
> >> >> > calculation),
> > but
> >> >> > if CPU is not there, could you help to explain how can we get the 
> >> >> > socket
> >> >> > information from it?
> >> >>
> >> >> Now a fundamental question is whether this was on a system where
> >> >> CPUs are actually hotpluggable - it's quite common for APIC IDs to be
> >> >> set to invalid values when the firmware just allocates a large table
> >> >> and only fills (and marks enabled) the slots for existing CPUs.
> >> >>
> >> >> In any event with sparse APIC ID allocations like this
> >> >>
> >> >> ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
> >> >> ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
> >> >> ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
> >> >> ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
> >> >> ACPI: LAPIC (acpi_id[0x10] lapic_id[0x20] enabled)
> >> >> ACPI: LAPIC (acpi_id[0x12] lapic_id[0x22] enabled)
> >> >> ACPI: LAPIC (acpi_id[0x14] lapic_id[0x24] enabled)
> >> >> ACPI: LAPIC (acpi_id[0x16] lapic_id[0x26] enabled)
> >> >>
> >> >> making assumptions about the socket ID range being a subset of
> >> >> the NR_CPUS one is wrong - you'll either need some data structure
> >> >> other than a flat array to deal with this, or at least dimension the
> >> >> thing using MAX_APICS (but you'll then realize that you end up
> >> >> with an even larger array, i.e. it'll become less acceptable). And
> >> >> yes, other users making invalid assumptions on the socket ID
> >> >> ranges should be fixed as well (i.e. likely you would want a cleanup
> >> >> patch first, and then put the your QoS stuff on top).
> >> >
> >> > From your above explanation, it seems that we still cannot get exact 
> >> > socket
> >> > number from the MADT table,
> >>
> >> Why not? The socket IDs are derived from the LAPIC IDs, and as long
> >> as you have the full set of the latter, you can also determine the full
> >> set of the former.
> >
> > I dumped the MADT table from the machine that supports CPU hotplug.
> > However if I remove some CPUs from the sockets, those disabled CPU's APIC
> ID
> > is shown as 0xFF.
> > Therefore to me it is difficult to get the full set of the LAPIC IDs, so as
> > the socket IDs.
> 
> Okay, that convinces me then that we need to deal with socket IDs
> being known only at hotplug time.

Thanks for the confirmation.

> 
> > If use radix tree, it is implemented as a list, where all nodes are
> > allocated by xmalloc(). It is difficult to share these radix nodes between
> > Xen and Dom0 tool stack. Besides, current libxl/libxc doesn't have such 
> > radix
> > APIs.
> 
> But you could get away with a split approach: Provide the socket ID
> list via ordinary hypercall means (if this isn't already derivable from
> the topology sysctl anyway), and only share the per-socket data
> page(s).

sysctl->u.getcqminfo has size limitation of 128 bytes.
The per-socket MFN array _may_ exceed this limitation...

We discussed this issue in previous threads, that's how this 2-level of 
data_mfn/data page sharing mechanism is proposed.

Thanks,
Dongxiao

> 
> Jan
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel