Re: [Xen-devel] [Linux 3.7-rc8] DomU: WARNING: at net/core/skbuff.c:3444 skb_try_coalesce+0x359/0x390()

[Sander Eikelenboom <linux@xxxxxxxxxxxxxx>]

Sunday, December 9, 2012, 10:36:02 PM, you wrote:


> Sunday, December 9, 2012, 4:06:37 PM, you wrote:

>> Hi Ian,

>> I guess this issue is similar with this one 
>> http://comments.gmane.org/gmane.linux.network/236358. And netfront also 
>> needs to reserve some tail room for IP/TCP headers too?

> Hi Annie,
> Thanks for digging this up !

> That looks indeed remarkable similar. It's probably revealed by the other 
> netfront/netback changes in 3.7, because i have never seen it before.
> It also seems to take some time before it gets triggered.
> Only the code in netfront.c is that much different, i don't think i'm able to 
> determine the proper size and suggest a fix.

Hi Ian,

> Why is this being discussed in private mail? Please can you resend to
> xen-devel and/or netdev.

Sorry i missed the CC to xen-devel on the first post, wasn't meant to be 
private in any way.

> I have a vague recollection of a patch to set skb->truesize more
> accurately in xennet_poll (netfront), but I can't seem to find any
> reference to it now.

I tried to gain some extra info from net/core/skbuff.c around the warn.

diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 3f0636c..a7831b7 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -3442,6 +3442,10 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff 
*from,
        }

        WARN_ON_ONCE(delta < len);
+       if(delta < len){
+                net_warn_ratelimited("netfront: WARN delta(%d) < len(%d) 
truesize(%d) SKB_DATA_ALIGN(%d)  SKB_TRUESIZE(%d) \n",
+                         delta,len,from->truesize,SKB_DATA_ALIGN(sizeof(struct 
sk_buff)),SKB_TRUESIZE(skb_end_offset(from)));
+       }

        memcpy(skb_shinfo(to)->frags + skb_shinfo(to)->nr_frags,
               skb_shinfo(from)->frags,



This results in:

[ 5557.333423] ------------[ cut here ]------------
[ 5557.333471] WARNING: at net/core/skbuff.c:3444 skb_try_coalesce+0x355/0x3d0()
[ 5557.333495] Modules linked in:
[ 5557.333519] Pid: 1872, comm: apache2 Not tainted 3.7.0-rc8-20121209-netdebug 
#1
[ 5557.333543] Call Trace:
[ 5557.333556]  <IRQ>  [<ffffffff8106752a>] warn_slowpath_common+0x7a/0xb0
[ 5557.333598]  [<ffffffff81067575>] warn_slowpath_null+0x15/0x20
[ 5557.333621]  [<ffffffff816a2b25>] skb_try_coalesce+0x355/0x3d0
[ 5557.333646]  [<ffffffff8175a049>] tcp_try_coalesce+0x69/0xc0
[ 5557.333669]  [<ffffffff8175a0f4>] tcp_queue_rcv+0x54/0x100
[ 5557.333691]  [<ffffffff8176023f>] ? tcp_transmit_skb+0x7ff/0x8d0
[ 5557.333714]  [<ffffffff8175ebbb>] tcp_rcv_established+0x2bb/0x6a0
[ 5557.333737]  [<ffffffff8176733f>] ? tcp_v4_rcv+0x6cf/0xb10
[ 5557.333758]  [<ffffffff81766925>] tcp_v4_do_rcv+0x135/0x480
[ 5557.333951]  [<ffffffff8180be02>] ? _raw_spin_lock_nested+0x42/0x50
[ 5557.333975]  [<ffffffff8176733f>] ? tcp_v4_rcv+0x6cf/0xb10
[ 5557.334009]  [<ffffffff817675cd>] tcp_v4_rcv+0x95d/0xb10
[ 5557.334032]  [<ffffffff810b21e8>] ? lock_acquire+0xd8/0x100
[ 5557.334055]  [<ffffffff81743d95>] ? ip_local_deliver_finish+0x45/0x230
[ 5557.334081]  [<ffffffff81743e6a>] ip_local_deliver_finish+0x11a/0x230
[ 5557.334105]  [<ffffffff81743d95>] ? ip_local_deliver_finish+0x45/0x230
[ 5557.334129]  [<ffffffff81743fb8>] ip_local_deliver+0x38/0x80
[ 5557.334152]  [<ffffffff8174357a>] ip_rcv_finish+0x15a/0x630
[ 5557.334175]  [<ffffffff81743c68>] ip_rcv+0x218/0x300
[ 5557.334197]  [<ffffffff816abf8d>] __netif_receive_skb+0x65d/0x8d0
[ 5557.334220]  [<ffffffff816aba75>] ? __netif_receive_skb+0x145/0x8d0
[ 5557.334244]  [<ffffffff810ae48d>] ? trace_hardirqs_on+0xd/0x10
[ 5557.334268]  [<ffffffff810fafc3>] ? free_hot_cold_page+0x1b3/0x1e0
[ 5557.334294]  [<ffffffff816ae4e8>] netif_receive_skb+0x28/0xf0
[ 5557.334315]  [<ffffffff816a4003>] ? __pskb_pull_tail+0x253/0x340
[ 5557.334343]  [<ffffffff814b3c75>] xennet_poll+0xad5/0xe10
[ 5557.334379]  [<ffffffff816af296>] net_rx_action+0x136/0x260
[ 5557.334403]  [<ffffffff8106f3c1>] ? __do_softirq+0x71/0x1a0
[ 5557.334426]  [<ffffffff8106f419>] __do_softirq+0xc9/0x1a0
[ 5557.334448]  [<ffffffff8180e97c>] call_softirq+0x1c/0x30
[ 5557.334470]  [<ffffffff8100fd95>] do_softirq+0x85/0xf0
[ 5557.334491]  [<ffffffff8106f28e>] irq_exit+0x9e/0xd0
[ 5557.334514]  [<ffffffff813463af>] xen_evtchn_do_upcall+0x2f/0x40
[ 5557.334537]  [<ffffffff8180e9de>] xen_do_hypervisor_callback+0x1e/0x30
[ 5557.334566]  <EOI>  [<ffffffff812b3d30>] ? copy_user_generic_string+0x30/0x40
[ 5557.334607]  [<ffffffff817543ea>] ? tcp_sendmsg+0xafa/0xe10
[ 5557.334632]  [<ffffffff8177a3c9>] ? inet_sendmsg+0xa9/0x100
[ 5557.334654]  [<ffffffff8177a320>] ? inet_autobind+0x70/0x70
[ 5557.334676]  [<ffffffff81697490>] ? sock_destroy_inode+0x40/0x40
[ 5557.334698]  [<ffffffff816975bd>] ? sock_aio_write+0x12d/0x140
[ 5557.334723]  [<ffffffff81144a0b>] ? do_sync_readv_writev+0x9b/0xe0
[ 5557.334750]  [<ffffffff811450bf>] ? do_readv_writev+0xcf/0x1d0
[ 5557.334785]  [<ffffffff811451fe>] ? vfs_writev+0x3e/0x60
[ 5557.334806]  [<ffffffff8114534a>] ? sys_writev+0x5a/0xc0
[ 5557.334826]  [<ffffffff812b537e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 5557.334856]  [<ffffffff8180d6e9>] ? system_call_fastpath+0x16/0x1b
[ 5557.334876] ---[ end trace 62036fd3c663553e ]---
[ 5557.335039] skbuff: netfront: WARN delta(18634) < len(18824) truesize(19530) 
SKB_DATA_ALIGN(256)  SKB_TRUESIZE(896)
[ 6195.800823] skbuff: netfront: WARN delta(18634) < len(18824) truesize(19530) 
SKB_DATA_ALIGN(256)  SKB_TRUESIZE(896)

I hope it gives some more insight.

--

Sander



> --
> Sander


>> Thanks
>> Annie

>> On 2012-12-9 4:14, Sander Eikelenboom wrote:
>>> Hi All,
>>>
>>> I still seem to hit some network warn in 3.7-rc8+.
>>> It only seems to appear in guests and not in dom0, it happens every once in 
>>> a while and in multiple guests.
>>>
>>> [  778.846089] ------------[ cut here ]------------
>>> [  778.846107] WARNING: at net/core/skbuff.c:3444 
>>> skb_try_coalesce+0x359/0x390()
>>> [  778.846114] Modules linked in:
>>> [  778.846121] Pid: 0, comm: swapper/0 Not tainted 3.7.0-rc8-20121208 #1
>>> [  778.846127] Call Trace:
>>> [  778.846131]  <IRQ>  [<ffffffff8106752a>] warn_slowpath_common+0x7a/0xb0
>>> [  778.846143]  [<ffffffff81067575>] warn_slowpath_null+0x15/0x20
>>> [  778.846149]  [<ffffffff816a2b29>] skb_try_coalesce+0x359/0x390
>>> [  778.846157]  [<ffffffff8175a009>] tcp_try_coalesce+0x69/0xc0
>>> [  778.846163]  [<ffffffff8175a0b4>] tcp_queue_rcv+0x54/0x100
>>> [  778.846168]  [<ffffffff817602ff>] ? tcp_wfree+0x2f/0x140
>>> [  778.846174]  [<ffffffff8175eb7b>] tcp_rcv_established+0x2bb/0x6a0
>>> [  778.846180]  [<ffffffff817672ff>] ? tcp_v4_rcv+0x6cf/0xb10
>>> [  778.846186]  [<ffffffff817668e5>] tcp_v4_do_rcv+0x135/0x480
>>> [  778.846192]  [<ffffffff8180bdc2>] ? _raw_spin_lock_nested+0x42/0x50
>>> [  778.846198]  [<ffffffff817672ff>] ? tcp_v4_rcv+0x6cf/0xb10
>>> [  778.846203]  [<ffffffff8176758d>] tcp_v4_rcv+0x95d/0xb10
>>> [  778.846209]  [<ffffffff810b21e8>] ? lock_acquire+0xd8/0x100
>>> [  778.846216]  [<ffffffff81743d55>] ? ip_local_deliver_finish+0x45/0x230
>>> [  778.846222]  [<ffffffff81743e2a>] ip_local_deliver_finish+0x11a/0x230
>>> [  778.846228]  [<ffffffff81743d55>] ? ip_local_deliver_finish+0x45/0x230
>>> [  778.846284]  [<ffffffff81743f78>] ip_local_deliver+0x38/0x80
>>> [  778.846291]  [<ffffffff8174353a>] ip_rcv_finish+0x15a/0x630
>>> [  778.846297]  [<ffffffff81743c28>] ip_rcv+0x218/0x300
>>> [  778.846303]  [<ffffffff816abf4d>] __netif_receive_skb+0x65d/0x8d0
>>> [  778.846309]  [<ffffffff816aba35>] ? __netif_receive_skb+0x145/0x8d0
>>> [  778.846315]  [<ffffffff810ae48d>] ? trace_hardirqs_on+0xd/0x10
>>> [  778.846322]  [<ffffffff810fafc3>] ? free_hot_cold_page+0x1b3/0x1e0
>>> [  778.846329]  [<ffffffff816ae4a8>] netif_receive_skb+0x28/0xf0
>>> [  778.846334]  [<ffffffff816a3fc3>] ? __pskb_pull_tail+0x253/0x340
>>> [  778.846342]  [<ffffffff814b3c75>] xennet_poll+0xad5/0xe10
>>> [  778.846349]  [<ffffffff816af256>] net_rx_action+0x136/0x260
>>> [  778.846355]  [<ffffffff8106f419>] __do_softirq+0xc9/0x1a0
>>> [  778.846361]  [<ffffffff8180e93c>] call_softirq+0x1c/0x30
>>> [  778.846368]  [<ffffffff8100fd95>] do_softirq+0x85/0xf0
>>> [  778.846373]  [<ffffffff8106f28e>] irq_exit+0x9e/0xd0
>>> [  778.846380]  [<ffffffff813463af>] xen_evtchn_do_upcall+0x2f/0x40
>>> [  778.846386]  [<ffffffff8180e99e>] xen_do_hypervisor_callback+0x1e/0x30
>>> [  778.846391]  <EOI>  [<ffffffff810013aa>] ? 
>>> xen_hypercall_sched_op+0xa/0x20
>>> [  778.846401]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
>>> [  778.846408]  [<ffffffff81008850>] ? xen_safe_halt+0x10/0x20
>>> [  778.846415]  [<ffffffff810170f0>] ? default_idle+0x40/0x90
>>> [  778.846421]  [<ffffffff810174a6>] ? cpu_idle+0x96/0xf0
>>> [  778.846428]  [<ffffffff817e4c0c>] ? rest_init+0xbc/0xd0
>>> [  778.846433]  [<ffffffff817e4b50>] ? csum_partial_copy_generic+0x170/0x170
>>> [  778.846441]  [<ffffffff81ee7be7>] ? start_kernel+0x390/0x39d
>>> [  778.846447]  [<ffffffff81ee7677>] ? repair_env_string+0x5b/0x5b
>>> [  778.846454]  [<ffffffff81ee7356>] ? x86_64_start_reservations+0x131/0x136
>>> [  778.846461]  [<ffffffff81eea915>] ? xen_start_kernel+0x54e/0x550
>>> [  778.846467] ---[ end trace d13d814dbabaca0e ]---
>>>
>>>
>>> --
>>> Sander
>>>
>>>   






_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel