Re: [Xen-devel] Problems with hyperthreading in Windows HVM

[Roderick Colenbrander <thunderbird2k@xxxxxxxxx>]

On Thu, Mar 1, 2012 at 8:22 AM, Ian Campbell <Ian.Campbell@xxxxxxxxxx> wrote:
> On Wed, 2012-02-29 at 18:13 +0000, Roderick Colenbrander wrote:
>> On Wed, Feb 29, 2012 at 2:37 PM, Ian Campbell <Ian.Campbell@xxxxxxxxxx> 
>> wrote:
>> > On Wed, 2012-02-29 at 00:28 +0000, Roderick Colenbrander wrote:
>> >> Hi,
>> >>
>> >> I have been trying to get Hyperthreading to work in a Windows HVM on
>> >> Xen 4.1.2 as described in 'xmexample.hvm'. I think I have set it up
>> >> correctly, but I can't seem to get it to work. There is not much
>> >> documentation on it and most topics are from years ago.
>> >
>> > I don't know the answers to your questions, or whether this particular
>> > type of cpuid modification has ever worked. But might it be that
>> > something on the hypervisor side is also modifying the cpuid results?
>>
>> It looks like there is code in xen/arch/x86/hvm/hvm.c which deals with
>> some more cpuid overrides. Initially it looked like libxc was doing
>> all the filtering and that 'XEN_DOMCTL_set_cpuid' just stored the
>> info. I'm not sure yet how that hypercall and the hvm.c code interact.
>> I have to dive more into it.
>>
>> It also seems that for modern CPUs cpuid function 11 contains
>> additional topology information which may have to patched as well.
>>
>> > I suppose you could start at xen/arch/x86/hvm/emulate.c:hvmemul_cpuid
>> > and trace down to the actual code in question? Looks like hvm_cpuid() is
>> > probably the key function of interest.
>> >
>> > Note sure where APIC IDs come from either, xen/arch/x86/hvm/vlapic.c
>> > might be an interesting starting point. Or perhaps this is done by
>> > hvmloader tools/firmware/hvmloader.c?
>> >
>> > More questions than answers in the above, sorry about that!
>> >
>> > Ian.
>> >
>>
>> I'm going to dive further into the code and look at the places you
>> pointed to. I guess it can be made to work with some patching.
>> Assuming this would all work, I really wonder whether how this
>> hyperthreading fixup would interact with Xen itself. Normally Xen
>> hides hyperthreading for a reason and each guest works with 'virtual
>> cores' which get scheduled. My feeling is that the hyperthreading
>> fixup would have to be combined with CPU pinning to the correct
>> physical cores. Then to get 4 logical cores in the VM, you need to
>> assign 2 VCPUs to the VM. Would the guest OS really be able to
>> schedule threads over 4 logical cores? I really don't know how this
>> would work in Xen.
>
> Normally Xen treats every HT as a CPU (roughly speaking) and will
> schedule VCPUs between them I think the scheduler does have some basic
> smarts about HT i.e. it won't schedule 2 VCPUs on sibling HTs while
> leaving another pair totally idle.
>
> Ultimately it depends on whether you want to actually expose HT to the
> guest or if you just want the guest to think it has HT. In your use case
> is having the workload scheduled on two sibling threads actually better
> than giving them a core each?

Let me describe the workload. It consists of closed-source Windows
applications which use multiple threads. Since each thread is not too
stressful (there is blocking I/O and other stuff involved)
hyperthreading helps quite a bit. The problem is that the applications
scale the work load based on the number of logical and physical cores
(through Win32 GetLogicalProcessorInformation and friends). Since on
Xen each VCPU is seen as a physical core, the work load the system can
handle is estimated incorrectly. In order for the workload to utilize
the system at best, hyperthreading would have to be exposed to the VM.

>
> If you want to really expose HT then you will probably have to play
> games with pinning as you suggest.
>
> Ian.
>

I will try to play with the cpuid options. I have written some test
applications based on Intel datasheets, but haven't had time to
continue with it yet. I will report back when I know more and document
how to get this to work.

Roderick

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel