[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Xen-unstable Linux 3.14-rc3 and 3.13 Network troubles




On 2014/2/20 19:18, Sander Eikelenboom wrote:
Thursday, February 20, 2014, 10:49:58 AM, you wrote:


On 2014/2/19 5:25, Sander Eikelenboom wrote:
Hi All,

I'm currently having some network troubles with Xen and recent linux kernels.

- When running with a 3.14-rc3 kernel in dom0 and a 3.13 kernel in domU
    I get what seems to be described in this thread: 
http://www.spinics.net/lists/netdev/msg242953.html

    In the guest:
    [57539.859584] net eth0: rx->offset: 0, size: 4294967295
    [57539.859599] net eth0: rx->offset: 0, size: 4294967295
    [57539.859605] net eth0: rx->offset: 0, size: 4294967295
    [57539.859610] net eth0: Need more slots
    [58157.675939] net eth0: Need more slots
    [58725.344712] net eth0: Need more slots
    [61815.849180] net eth0: rx->offset: 0, size: 4294967295
    [61815.849205] net eth0: rx->offset: 0, size: 4294967295
    [61815.849216] net eth0: rx->offset: 0, size: 4294967295
    [61815.849225] net eth0: Need more slots
This issue is familiar... and I thought it get fixed.
  From original analysis for similar issue I hit before, the root cause
is netback still creates response when the ring is full. I remember
larger MTU can trigger this issue before, what is the MTU size?
In dom0 both for the physical nics and the guest vif's MTU=1500
In domU the eth0 also has MTU=1500.

So it's not jumbo frames .. just everywhere the same plain defaults ..

With the patch from Wei that solves the other issue, i'm still seeing the Need 
more slots issue on 3.14-rc3+wei's patch now.
I have extended the "need more slots warn" to also print the cons, slots, max,  
rx->offset, size, hope that gives some more insight.
But it indeed is the VM were i had similar issues before, the primary thing 
this VM does is 2 simultaneous rsync's (one push one pull) with some gigabytes 
of data.

This time it was also acompanied by a "grant_table.c:1857:d0 Bad grant reference 
" as seen below, don't know if it's a cause or a effect though.

The log "grant_table.c:1857:d0 Bad grant reference " was also seen before.
Probably the response overlaps the request and grantcopy return error when using wrong grant reference, Netback returns resp->status with ||XEN_NETIF_RSP_ERROR(-1) which is 4294967295 printed above from frontend. Would it be possible to print log in xenvif_rx_action of netback to see whether something wrong with max slots and used slots?

Thanks
Annie


Will keep you posted when it triggers again with the extra info in the warn.

--
Sander



Thanks
Annie
    Xen reports:
    (XEN) [2014-02-18 03:22:47] grant_table.c:1857:d0 Bad grant reference 
19791875
    (XEN) [2014-02-18 03:42:33] grant_table.c:1857:d0 Bad grant reference 
268435460
    (XEN) [2014-02-18 04:15:23] grant_table.c:289:d0 Increased maptrack size to 
14 frames
    (XEN) [2014-02-18 04:15:27] grant_table.c:289:d0 Increased maptrack size to 
15 frames
    (XEN) [2014-02-18 04:15:48] grant_table.c:289:d0 Increased maptrack size to 
16 frames
    (XEN) [2014-02-18 04:15:50] grant_table.c:289:d0 Increased maptrack size to 
17 frames
    (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 
18 frames
    (XEN) [2014-02-18 04:15:55] grant_table.c:289:d0 Increased maptrack size to 
19 frames
    (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 
20 frames
    (XEN) [2014-02-18 04:15:56] grant_table.c:289:d0 Increased maptrack size to 
21 frames
    (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 
22 frames
    (XEN) [2014-02-18 04:15:59] grant_table.c:289:d0 Increased maptrack size to 
23 frames
    (XEN) [2014-02-18 04:16:00] grant_table.c:289:d0 Increased maptrack size to 
24 frames
    (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 
25 frames
    (XEN) [2014-02-18 04:16:05] grant_table.c:289:d0 Increased maptrack size to 
26 frames
    (XEN) [2014-02-18 04:16:06] grant_table.c:289:d0 Increased maptrack size to 
27 frames
    (XEN) [2014-02-18 04:16:12] grant_table.c:289:d0 Increased maptrack size to 
28 frames
    (XEN) [2014-02-18 04:16:18] grant_table.c:289:d0 Increased maptrack size to 
29 frames
    (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 
268435460
    (XEN) [2014-02-18 04:17:00] grant_table.c:1857:d0 Bad grant reference 
268435460
    (XEN) [2014-02-18 04:34:03] grant_table.c:1857:d0 Bad grant reference 
4325377



Another issue with networking is when running both dom0 and domU's with a 
3.14-rc3 kernel:
    - i can ping the guests from dom0
    - i can ping dom0 from the guests
    - But i can't ssh or access things by http
    - I don't see any relevant error messages ...
    - This is with the same system and kernel config as with the 3.14 and 3.13 
combination above
      (that previously worked fine)

--

Sander


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel




_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.