Re: [Xen-devel] compound skb frag pages appearing in start_xmit

[Sander Eikelenboom <linux@xxxxxxxxxxxxxx>]

Thursday, October 11, 2012, 10:02:26 AM, you wrote:

> On Wed, 2012-10-10 at 15:49 +0100, Sander Eikelenboom wrote:
>> Wednesday, October 10, 2012, 3:09:58 PM, you wrote:
>> 
>> > On Wed, 2012-10-10 at 11:13 +0100, Ian Campbell wrote:
>> >> I haven't tackled netfront yet. 
>> 
>> > I seem to be totally unable to reproduce the equivalent issue on the
>> > netfront xmit side, even though it seems like the loop in
>> > xennet_make_frags ought to be obviously susceptible to it.
>> 
>> > Konrad, Sander, are either of you able to repro, e.g. with:
>> 
>> 
>> Hmrrrmm i don't see any traces, only strange behaviour ..
>> 
>> - i can connect to guests by ssh, but it's sluggish, and sometimes stops 
>> working

> I saw something like this (ssh sluggish) even with dom0 itself. I'm
> trying to see if I can characterise it enough to reliably bisect it.

> I already switched out xen-unstable for 4.2-testing but that didn't make
> any difference.



>> - The guest seem to keep trying to connect to netback:
>> 
>> [  658.276719] xen_bridge: port 2(vif40.0) entered forwarding state
>> [  658.282258] xen_bridge: port 2(vif40.0) entered forwarding state
>> [  663.945964] xen_bridge: port 7(vif39.0) entered forwarding state
>> [  669.674277] xen_bridge: port 2(vif40.0) entered disabled state
>> [  669.680290] device vif40.0 left promiscuous mode
>> [  669.685464] xen_bridge: port 2(vif40.0) entered disabled state
>> [  672.857222] device vif41.0 entered promiscuous mode
>> [  673.166254] xen-blkback:ring-ref 8, event-channel 9, protocol 1 
>> (x86_64-abi)
>> [  673.176368] xen_bridge: port 2(vif41.0) entered forwarding state
>> [  673.182042] xen_bridge: port 2(vif41.0) entered forwarding state
>> [  674.439725] xen_bridge: port 7(vif39.0) entered disabled state
>> [  674.445708] device vif39.0 left promiscuous mode
>> [  674.450955] xen_bridge: port 7(vif39.0) entered disabled state
>> [  677.726040] device vif42.0 entered promiscuous mode
>> [  678.053381] xen-blkback:ring-ref 8, event-channel 9, protocol 1 
>> (x86_64-abi)
>> [  678.062804] xen_bridge: port 7(vif42.0) entered forwarding state
>> [  678.068433] xen_bridge: port 7(vif42.0) entered forwarding state
>> [  688.224736] xen_bridge: port 2(vif41.0) entered forwarding state
>> [  693.080557] xen_bridge: port 7(vif42.0) entered forwarding state
>> [  700.786276] xen_bridge: port 7(vif42.0) entered disabled state
>> [  700.792484] device vif42.0 left promiscuous mode
>> [  700.802409] xen_bridge: port 7(vif42.0) entered disabled state
>> [  704.133606] device vif43.0 entered promiscuous mode
>> [  704.460160] xen-blkback:ring-ref 8, event-channel 9, protocol 1 
>> (x86_64-abi)
>> [  704.469800] xen_bridge: port 7(vif43.0) entered forwarding state
>> [  704.475303] xen_bridge: port 7(vif43.0) entered forwarding state
>> [  719.493788] xen_bridge: port 7(vif43.0) entered forwarding state
>> [  726.302456] xen_bridge: port 7(vif43.0) entered disabled state
>> [  726.308898] device vif43.0 left promiscuous mode
>> [  726.314029] xen_bridge: port 7(vif43.0) entered disabled state
>> 
>> All the guests are already up, but this keeps on going and going and going 
>> ....

> The domain number seems to be climbing, are you sure something isn't
> (crashing and) restarting?

Probably due to the BUG_ON from the patch below, i changed it into a WARN_ON.
And i seem to hit it, but only in one of the guests at the moment and it 
triggers quite irregularly.

[   34.298549] ------------[ cut here ]------------
[   34.298567] WARNING: at drivers/net/xen-netfront.c:465 
xennet_start_xmit+0x7fe/0x860()
[   34.298574] Modules linked in:
[   34.298597] Pid: 1580, comm: sshd Not tainted 3.6.0pre-rc1-20121011 #1
[   34.298603] Call Trace:
[   34.298611]  [<ffffffff810664ea>] warn_slowpath_common+0x7a/0xb0
[   34.298617]  [<ffffffff81066535>] warn_slowpath_null+0x15/0x20
[   34.298623]  [<ffffffff8146d89e>] xennet_start_xmit+0x7fe/0x860
[   34.298631]  [<ffffffff8161f349>] dev_hard_start_xmit+0x209/0x460
[   34.298637]  [<ffffffff8163b036>] sch_direct_xmit+0xf6/0x290
[   34.298643]  [<ffffffff8161f746>] dev_queue_xmit+0x1a6/0x5a0
[   34.298649]  [<ffffffff8161f5a0>] ? dev_hard_start_xmit+0x460/0x460
[   34.298656]  [<ffffffff810aa8e5>] ? trace_softirqs_off+0x85/0x1b0
[   34.298663]  [<ffffffff816b9536>] ip_finish_output+0x226/0x530
[   34.298668]  [<ffffffff816b93dd>] ? ip_finish_output+0xcd/0x530
[   34.298674]  [<ffffffff816b9899>] ip_output+0x59/0xe0
[   34.298680]  [<ffffffff816b83b8>] ip_local_out+0x28/0x90
[   34.298687]  [<ffffffff816b896f>] ip_queue_xmit+0x17f/0x4a0
[   34.298692]  [<ffffffff816b87f0>] ? ip_send_unicast_reply+0x340/0x340
[   34.298699]  [<ffffffff810a0ba7>] ? getnstimeofday+0x47/0xe0
[   34.298705]  [<ffffffff8160f4c9>] ? __skb_clone+0x29/0x120
[   34.298711]  [<ffffffff816cea20>] tcp_transmit_skb+0x400/0x8d0
[   34.298717]  [<ffffffff816d19fa>] tcp_write_xmit+0x21a/0xa50
[   34.298723]  [<ffffffff816d225b>] tcp_push_one+0x2b/0x40
[   34.298728]  [<ffffffff816c2dec>] tcp_sendmsg+0x8dc/0xe20
[   34.298735]  [<ffffffff816e8f19>] inet_sendmsg+0xa9/0x100
[   34.298740]  [<ffffffff816e8e70>] ? inet_autobind+0x70/0x70
[   34.298746]  [<ffffffff810b0f88>] ? lock_acquire+0xd8/0x100
[   34.298753]  [<ffffffff8160630d>] sock_aio_write+0x12d/0x140
[   34.298762]  [<ffffffff811435b2>] do_sync_write+0xa2/0xe0
[   34.298768]  [<ffffffff810ad22d>] ? trace_hardirqs_on+0xd/0x10
[   34.298774]  [<ffffffff811441d4>] vfs_write+0x174/0x190
[   34.298779]  [<ffffffff811442fa>] sys_write+0x5a/0xa0
[   34.298786]  [<ffffffff812b33de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[   34.298792]  [<ffffffff817491cc>] cstar_dispatch+0x7/0x26
[   34.298797] ---[ end trace 2e28eec93b7a8b74 ]---


Complete dmesg from guest attached.



>> > diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
>> > index b06ef81..8a3f770 100644
>> > --- a/drivers/net/xen-netfront.c
>> > +++ b/drivers/net/xen-netfront.c
>> > @@ -462,6 +462,8 @@ static void xennet_make_frags(struct sk_buff *skb, 
>> > struct net_device *dev,
>> >                 ref = gnttab_claim_grant_reference(&np->gref_tx_head);
>> >                 BUG_ON((signed short)ref < 0);
>> >  
>> > +               BUG_ON(PageCompound(skb_frag_page(frag)));
>> > +
>> >                 mfn = pfn_to_mfn(page_to_pfn(skb_frag_page(frag)));
>> >                 gnttab_grant_foreign_access_ref(ref, 
>> > np->xbdev->otherend_id,
>> >                                                 mfn, GNTMAP_readonly);
>> 
>> > My repro for netback was just to netcat a wodge of data from dom0->domU
>> > but going the other way doesn't seem to trigger.
>> 
>> 
>>

Attachment: dmesg-netfront.txt
Description: Text document

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel