Re: [Xen-devel] [Xen-users] FreeBSD PVHVM call for testing

[Matt Wilson <msw@xxxxxxxxxx>]

On Thu, May 30, 2013 at 10:31:43AM +0100, Ian Campbell wrote:
> On Wed, 2013-05-29 at 15:10 -0700, Matt Wilson wrote:
> > On Wed, May 29, 2013 at 06:25:56PM +0100, Ian Campbell wrote:
> > > On Wed, 2013-05-29 at 19:03 +0200, Roger Pau Monné wrote:
> > > > 
> > > > Since I'm not able to reproduce the cpuid != acpi_id case, could you
> > > > give it a try and report the results?
> > > 
> > > Konrad,
> > > 
> > > Might this same problem be related to the issue you saw doing PVHVM VCPU
> > > hotplug which you mentioned the other day?
> > >
> > > In general for HVM I suppose there isn't a strict relationship between
> > > the CPU number the kernel chooses for a CPU (which is somewhat
> > > arbitrarily up to the kernel) and the hypervisors VCPU number (which is
> > > exposed via the virtual APIC ID).
> > 
> > The CPU enumeration should follow the ACPI ID order in MADT (or
> > MP-table), right?

> You mean the guest's enumeration?

Right.

> I'd have thought that was entirely up to the guest. Using the APIC
> ID might be a reasonable implementation but there are surely others
> (e.g. collapsing sparse LAPIC IDs into contiguous cpu ids, using the
> top bits for NUMA node, etc etc).

On bare metal x86 Linux, the kernel enumerates CPUs based on an order
defined by the BIOS. Typically this means that all the cores are
enumerated first, followed by logical processors (HT/SMT). For Linux,
maxcpus=N/2 should disable HT on systems that enumerate processors in
the recommended order. Some history:
  https://bugzilla.kernel.org/show_bug.cgi?id=2317

> >   The local APIC ID (returned by cpuid
> > EAX=0000_0001h => EBX bits 31:24) shouldn't be used for
> > enumeration. Under Xen, the LAPIC ID is hardcoded to vCPU idx * 2:
> > 
> > void hvm_cpuid(unsigned int input, unsigned int *eax, unsigned int *ebx,
> > [...]
> >      case 0x1:
> >          /* Fix up VLAPIC details. */
> >          *ebx &= 0x00FFFFFFu;
> >          *ebx |= (v->vcpu_id * 2) << 24;
> > [...]
> >      case 0xb:
> >          /* Fix the x2APIC identifier. */
> >          *edx = v->vcpu_id * 2;
> >          break;
> > 
> > This isn't the way it works on some of our EC2 instances.  On our high
> > performance instance types, we provide x2APIC IDs that distinguish
> > threads, cores and sockets to provide the guest OS with CPU topology
> > information. The OS and applications can use CPUID EAX=0000_000Bh,
> > ECX=level (HT=0, Core=1, Socket=2) => EAX bits 4:0 to determine the
> > topology.
> 
> Hrm, that does beg the question of how an HVM guest is supposed to
> determine what Xen's idea of the VCPU number is for a given CPU.

The virtual BIOS provides both ACPI tables and a legacy MP-table that
gives the LAPIC id mapping. The guest could infer the Xen vCPU ID from
a processor's position in these tables. Or we could add a VCPUOP that
an enlightened guest could use to get the information more directly.

One question: why does a hypercall take a parameter that only has one
valid value? That value can be determined by looking at the current
running vCPU.

> I can't see anything which reverses that * 2 on the vcpu_op path and I
> don't suppose you implemented the inverse of your scheme.

The *2 is just for assigning the LAPIC ID, and I'm pretty sure that
Linux is assigning processor IDs sequentially at ACPI parse time.

--msw

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel