[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] netfront/netback multiqueue exhausting grants
On 21/01/16 12:19, Ian Campbell wrote: > On Thu, 2016-01-21 at 10:56 +0000, David Vrabel wrote: >> On 20/01/16 12:23, Ian Campbell wrote: >>> There have been a few reports recently[0] which relate to a failure of >>> netfront to allocate sufficient grant refs for all the queues: >>> >>> [ 0.533589] xen_netfront: can't alloc rx grant refs >>> [ 0.533612] net eth0: only created 31 queues >>> >>> Which can be worked around by increasing the number of grants on the >>> hypervisor command line or by limiting the number of queues permitted >>> by >>> either back or front using a module param (which was broken but is now >>> fixed on both sides, but I'm not sure it has been backported everywhere >>> such that it is a reliable thing to always tell users as a workaround). >>> >>> Is there any plan to do anything about the default/out of the box >>> experience? Either limiting the number of queues or making both ends >>> cope >>> more gracefully with failure to create some queues (or both) might be >>> sufficient? >>> >>> I think the crash after the above in the first link at [0] is fixed? I >>> think that was the purpose of ca88ea1247df "xen-netfront: update >>> num_queues >>> to real created" which was in 4.3. >> >> I think the correct solution is to increase the default maximum grant >> table size. > > That could well make sense, but then there will just be another higher > limit, so we should perhaps do both. > > i.e. factoring in: > * performance i.e. ability for N queues to saturate whatever sort of link > contemporary Linux can saturate these days, plus some headroom, or > whatever other ceiling seems sensible) > * grant table resource consumption i.e. (sensible max number of blks * nr > gnts per blk + sensible max number of vifs * nr gnts per vif + other > devs needs) < per guest grant limit) to pick both the default gnttab > size and the default max queuers. Yes. >> Although, unless you're using the not-yet-applied per-cpu rwlock patches >> multiqueue is terrible on many (multisocket) systems and the number of >> queue should be limited in netback to 4 or even just 2. > > Presumably the guest can't tell, so it can't do this. > > I think when you say "terrible" you don't mean "worse than without mq" but > rather "not realising the expected gains from a larger nunber of queues", > right?. Malcolm did the analysis but if I remember correctly, 8 queues performed about the same as 1 queue and 16 were worse than 1 queue. David _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |