RE: [Xen-devel] VT-d scalability issue

["Ian Pratt" <Ian.Pratt@xxxxxxxxxxxxx>]

> > But how much of the degradation is due to IOTLB pressure and how
much
> > is due to vcpu pinning?  If vcpu pinning doesn't give you much then
> > why add the automatic pinning just to get a little improvement on
> > older CPUs hooked up to a VT-d chipset?
> 
> Say, throughput of 1 pass-through domain is 100%,
> if not pin vcpu, average throughput of 8 pass-through domain is 59%.
> If pin vcpu, average is 95%.
> 
> So you can see how much vcpu pinning contribute to the performance.

For comparison, what are the results if you use a penryn with wbinvd
exit support?

Thanks,
Ian


 
> >
> >     eSk
> >
> >
> > > Randy (Weidong)
> >
> > >>
> > >> [And talking of IOTLB pressure, why can't Intel document the
IOTLB
> > >> sizes in the chipset docs?  Or even better, why can't these
values
> be
> > >> queried from the chipset?]
> > >>
> > >> eSk
> > >>
> > >>
> > >> [Edwin Zhai]
> > >>> Keir,
> > >>> I have found a VT-d scalability issue and want to some feed
> backs.
> > >>
> > >>> When I assign a pass-through NIC to a linux VM and increase the
> num
> > >>> of VMs, the iperf throughput for each VM drops greatly. Say,
> start 8
> > >>> VM running on a machine with 8 physical cpus, start 8 iperf
> client
> > >>> to connect each of them, the final result is only 60% of 1 VM.
> > >>
> > >>> Further investigation shows vcpu migration cause "cold" cache
for
> > >>> pass-through domain.  following code in vmx_do_resume try to
> > >>> invalidate orig processor's cache when 14 migration if this
> domain
> > >>> has pass-through device and no support for wbinvd vmexit.
> > >>
> > >>> 16 if ( has_arch_pdevs(v->domain) && !cpu_has_wbinvd_exiting ) {
> > >>> int cpu = v->arch.hvm_vmx.active_cpu;
> > >>> if ( cpu != -1 )
> > >>> on_selected_cpus(cpumask_of_cpu(cpu), wbinvd_ipi, NULL, 1,
> > >>
> > >>> }
> > >>
> > >>> So we want to pin vcpu to free processor for domains with
> > >>> pass-through device in creation process, just like what we did
> for
> > >>> NUMA system.
> > >>
> > >>> What do you think of it? Or have other ideas?
> > >>
> > >>> Thanks,
> > >>
> > >>
> > >>> --
> > >>> best rgds,
> > >>> edwin
> > >>
> > >>> _______________________________________________
> > >>> Xen-devel mailing list
> > >>> Xen-devel@xxxxxxxxxxxxxxxxxxx
> > >>> http://lists.xensource.com/xen-devel
> > >>
> > >>
> > >> _______________________________________________
> > >> Xen-devel mailing list
> > >> Xen-devel@xxxxxxxxxxxxxxxxxxx
> > >> http://lists.xensource.com/xen-devel
> >
> >
> 
> --
> best rgds,
> edwin
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel