[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] large packet support in netfront driver and guest network throughput
On Sep 18, 2013, at 8:48 AM, Wei Liu <wei.liu2@xxxxxxxxxx> wrote: > On Tue, Sep 17, 2013 at 05:53:43PM +0000, Anirban Chakraborty wrote: >> >> >> On 9/17/13 1:25 AM, "Wei Liu" <wei.liu2@xxxxxxxxxx> wrote: >> >>> On Tue, Sep 17, 2013 at 10:09:21AM +0800, annie li wrote: >>>> <snip> >>>>>> I tried dom0 to dom0 and I got 9.4 Gbps, which is what I expected >>>>>> (with GRO turned on in the physical interface). However, when I run >>>>>> guest to guest, things fall off. Is large packet not supported in >>>>>> netfront? I thought otherwise. I looked at the code and I do not see >>>>>> any call to napi_gro_receive(), rather it is using >>>>>> netif_receive_skb(). netback seems to be sending GSO packets to the >>>>>> netfront, but it is being segmented to 1500 byte (as it appears from >>>>>> the tcpdump). >>>>>> >>>>> OK, I get your problem. >>>>> >>>>> Indeed netfront doesn't make use of GRO API at the moment. >>>> >>>> This is true. >>>> But I am wondering why large packet is not segmented into mtu size >>>> with upstream kernel? I did see large packets with upsteam kernel on >>>> receive guest(test between 2 domus on same host). >>>> >>> >>> I think Anirban's setup is different. The traffic is from a DomU on >>> another host. >>> >>> I will need to setup testing environment with 10G link to test this. >>> >>> Anirban, can you share your setup, especially DomU kernel version, are >>> you using upstream kernel in DomU? >> >> Sure.. >> I have two hosts, say h1 and h2 running XenServer 6.1. >> h1 running Centos 6.4, 64bit kernel, say guest1 and h2 running identical >> guest, guest2. >> > > Do you have exact version of your DomU' kernel? Is it available > somewhere online? Yes, it is 2.6.32-358.el6.x86_64. Sorry, I missed it out last time. > >> iperf server is running on guest1 with iperf client connecting from guest2. >> >> I haven't tried with upstream kernel yet. However, what I found out is >> that the netback on the receiving host is transmitting GSO segments to the >> guest (guest1), but the packets are segmented at the netfront interface. >> > > I just tried, with vanilla upstream kernel I can see large packet size > on DomU's side. > > I also tried to convert netfront to use GRO API (hopefully I didn't get > it wrong), I didn't see much improvement -- it's quite obvious because I > already saw large packet even without GRO. > > If you fancy trying GRO API, see attached patch. Note that you might > need to do some contextual adjustment as this patch is for upstream > kernel. > > Wei. > > ---8<--- > From ca532dd11d7b8f5f8ce9d2b8043dd974d9587cb0 Mon Sep 17 00:00:00 2001 > From: Wei Liu <wei.liu2@xxxxxxxxxx> > Date: Wed, 18 Sep 2013 16:46:23 +0100 > Subject: [PATCH] xen-netfront: convert to GRO API > > Signed-off-by: Wei Liu <wei.liu2@xxxxxxxxxx> > --- > drivers/net/xen-netfront.c | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c > index 36808bf..dd1011e 100644 > --- a/drivers/net/xen-netfront.c > +++ b/drivers/net/xen-netfront.c > @@ -952,7 +952,7 @@ static int handle_incoming_queue(struct net_device *dev, > u64_stats_update_end(&stats->syncp); > > /* Pass it up. */ > - netif_receive_skb(skb); > + napi_gro_receive(&np->napi, skb); > } > > return packets_dropped; > @@ -1051,6 +1051,8 @@ err: > if (work_done < budget) { > int more_to_do = 0; > > + napi_gro_flush(napi, false); > + > local_irq_save(flags); > > RING_FINAL_CHECK_FOR_RESPONSES(&np->rx, more_to_do); > -- > 1.7.10.4 I was able to see a bit of improvement (from 2.65 to 3.6 Gbps) with the following patch (your patch plus the advertisement of NETIF_F_GRO) : --------- diff --git a/xen-netfront.c.orig b/xen-netfront.c index 23e467d..bc673d3 100644 --- a/xen-netfront.c.orig +++ b/xen-netfront.c @@ -818,6 +818,7 @@ static int handle_incoming_queue(struct net_device *dev, { int packets_dropped = 0; struct sk_buff *skb; + struct netfront_info *np = netdev_priv(dev); while ((skb = __skb_dequeue(rxq)) != NULL) { struct page *page = NETFRONT_SKB_CB(skb)->page; @@ -846,7 +847,7 @@ static int handle_incoming_queue(struct net_device *dev, dev->stats.rx_bytes += skb->len; /* Pass it up. */ - netif_receive_skb(skb); + napi_gro_receive(&np->napi, skb); } return packets_dropped; @@ -981,6 +982,7 @@ err: if (work_done < budget) { int more_to_do = 0; + napi_gro_flush(napi); local_irq_save(flags); RING_FINAL_CHECK_FOR_RESPONSES(&np->rx, more_to_do); @@ -1182,7 +1184,8 @@ static struct net_device * __devinit xennet_create_dev(struct xenbus_device *dev netif_napi_add(netdev, &np->napi, xennet_poll, 64); /* Assume all features and let xennet_set_features fix up. */ - netdev->features = NETIF_F_IP_CSUM | NETIF_F_SG | NETIF_F_TSO; + netdev->features = NETIF_F_IP_CSUM | NETIF_F_SG | NETIF_F_TSO | + NETIF_F_GRO; SET_ETHTOOL_OPS(netdev, &xennet_ethtool_ops); SET_NETDEV_DEV(netdev, &dev->dev); ----------- tcpdump showed that the guest interface received large packets. I haven't checked upstream kernel as guest though. Anirban _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |