Re: [Xen-devel] null domains after xl destroy

[Juergen Gross <jgross@xxxxxxxx>]

On 03/05/17 12:45, Steven Haigh wrote:
> Just wanted to give this a little nudge now people seem to be back on
> deck...

Things seem to be more complicated than I thought.

There are clearly paths leading to use-after-free scenarios, e.g. the
one of the backtrace below:

xen_blkbk_remove() will free be regardless of the return value of
xen_blkif_disconnect(). If -EBUSY is returned xen_blkif_disconnect()
will be called again at the end of an I/O still in progress, leading
to a call of xenbus_unmap_ring_vfree(blkif->be->dev, ...) with be
already freed...

As Roger already said: this is a complete mess.

Working on a patch now...

BTW: Glenn, the debug patch isn't important any longer. It was just
meant to locate the problem which is now known.


Juergen

> 
> On 01/05/17 10:55, Glenn Enright wrote:
>> On 19/04/17 22:09, Juergen Gross wrote:
>>> On 19/04/17 09:16, Roger Pau Monné wrote:
>>>> On Wed, Apr 19, 2017 at 06:39:41AM +0200, Juergen Gross wrote:
>>>>> On 19/04/17 03:02, Glenn Enright wrote:
>>>>>> Thanks Juergen. I applied that, to our 4.9.23 dom0 kernel, which still
>>>>>> shows the issue. When replicating the leak I now see this trace (via
>>>>>> dmesg). Hopefully that is useful.
>>>>>>
>>>>>> Please note, I'm going to be offline next week, but am keen to keep on
>>>>>> with this, it may just be a while before I followup is all.
>>>>>>
>>>>>> Regards, Glenn
>>>>>> http://rimuhosting.com
>>>>>>
>>>>>>
>>>>>> ------------[ cut here ]------------
>>>>>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:508
>>>>>> xen_blkbk_remove+0x138/0x140
>>>>>> Modules linked in: xen_pciback xen_netback xen_gntalloc xen_gntdev
>>>>>> xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4
>>>>>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state
>>>>>> iptable_security
>>>>>> iptable_raw igle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
>>>>>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp
>>>>>> llc
>>>>>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw sg i2c_i801
>>>>>> i2c_smbus
>>>>>> i2c_core e1000e ptp p000_edac edac_core raid1 sd_mod ahci libahci
>>>>>> floppy
>>>>>> dm_mirror dm_region_hash dm_log dm_mod
>>>>>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.23-1.el6xen.x86_64 #1
>>>>>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007
>>>>>>  ffffc90040cfbba8 ffffffff8136b61f 0000000000000013 0000000000000000
>>>>>>  0000000000000000 0000000000000000 ffffc90040cfbbf8 ffffffff8108007d
>>>>>>  ffffea0001373fe0 000001fc33394434 ffff880000000001 ffff88004d93fac0
>>>>>> Call Trace:
>>>>>>  [<ffffffff8136b61f>] dump_stack+0x67/0x98
>>>>>>  [<ffffffff8108007d>] __warn+0xfd/0x120
>>>>>>  [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20
>>>>>>  [<ffffffff814ebde8>] xen_blkbk_remove+0x138/0x140
>>>>>>  [<ffffffff814497f7>] xenbus_dev_remove+0x47/0xa0
>>>>>>  [<ffffffff814bcfd4>] __device_release_driver+0xb4/0x160
>>>>>>  [<ffffffff814bd0ad>] device_release_driver+0x2d/0x40
>>>>>>  [<ffffffff814bbfd4>] bus_remove_device+0x124/0x190
>>>>>>  [<ffffffff814b93a2>] device_del+0x112/0x210
>>>>>>  [<ffffffff81448113>] ? xenbus_read+0x53/0x70
>>>>>>  [<ffffffff814b94c2>] device_unregister+0x22/0x60
>>>>>>  [<ffffffff814ed7cd>] frontend_changed+0xad/0x4c0
>>>>>>  [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>>>>  [<ffffffff81449b57>] xenbus_otherend_changed+0xc7/0x140
>>>>>>  [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>>>>>  [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>>>>  [<ffffffff81449fe0>] frontend_changed+0x10/0x20
>>>>>>  [<ffffffff814477fc>] xenwatch_thread+0x9c/0x140
>>>>>>  [<ffffffff810bffa0>] ? woken_wake_function+0x20/0x20
>>>>>>  [<ffffffff816ed93a>] ? schedule+0x3a/0xa0
>>>>>>  [<ffffffff816f1436>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>>>>>  [<ffffffff810c0c5d>] ? complete+0x4d/0x60
>>>>>>  [<ffffffff81447760>] ? split+0xf0/0xf0
>>>>>>  [<ffffffff810a051d>] kthread+0xcd/0xf0
>>>>>>  [<ffffffff810a974e>] ? schedule_tail+0x1e/0xc0
>>>>>>  [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>>>>>  [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>>>>>  [<ffffffff816f1b45>] ret_from_fork+0x25/0x30
>>>>>> ---[ end trace ee097287c9865a62 ]---
>>>>>
>>>>> Konrad, Roger,
>>>>>
>>>>> this was triggered by a debug patch in xen_blkbk_remove():
>>>>>
>>>>>     if (be->blkif)
>>>>> -        xen_blkif_disconnect(be->blkif);
>>>>> +        WARN_ON(xen_blkif_disconnect(be->blkif));
>>>>>
>>>>> So I guess we need something like xen_blk_drain_io() in case of
>>>>> calls to
>>>>> xen_blkif_disconnect() which are not allowed to fail (either at the
>>>>> call
>>>>> sites of xen_blkif_disconnect() or in this function depending on a new
>>>>> boolean parameter indicating it should wait for outstanding I/Os).
>>>>>
>>>>> I can try a patch, but I'd appreciate if you could confirm this
>>>>> wouldn't
>>>>> add further problems...
>>>>
>>>> Hello,
>>>>
>>>> Thanks for debugging this, the easiest solution seems to be to
>>>> replace the
>>>> ring->inflight atomic_read check in xen_blkif_disconnect with a call to
>>>> xen_blk_drain_io instead, and making xen_blkif_disconnect return void
>>>> (to
>>>> prevent further issues like this one).
>>>
>>> Glenn,
>>>
>>> can you please try the attached patch (in dom0)?
>>>
>>>
>>> Juergen
>>>
>>
>> (resending with full CC list)
>>
>> I'm back. After testing unfortunately I'm still seeing the leak. The
>> below trace is with the debug patch applied as well under 4.9.25. It
>> looks very similar to me. I am still able to replicate this reliably.
>>
>> Regards, Glenn
>> http://rimuhosting.com
>>
>> ------------[ cut here ]------------
>> WARNING: CPU: 0 PID: 19 at drivers/block/xen-blkback/xenbus.c:511
>> xen_blkbk_remove+0x138/0x140
>> Modules linked in: ebt_ip xen_pciback xen_netback xen_gntalloc
>> xen_gntdev xen_evtchn xenfs xen_privcmd xt_CT ipt_REJECT nf_reject_ipv4
>> ebtable_filter ebtables xt_hashlimit xt_recent xt_state iptable_security
>> iptable_raw iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
>> nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables bridge stp llc
>> ipv6 crc_ccitt ppdev parport_pc parport serio_raw i2c_i801 i2c_smbus
>> i2c_core sg e1000e ptp pps_core i3000_edac edac_core raid1 sd_mod ahci
>> libahci floppy dm_mirror dm_region_hash dm_log dm_mod
>> CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.9.25-1.el6xen.x86_64 #1
>> Hardware name: Supermicro PDSML/PDSML+, BIOS 6.00 08/27/2007
>>  ffffc90040cfbb98 ffffffff8136b76f 0000000000000013 0000000000000000
>>  0000000000000000 0000000000000000 ffffc90040cfbbe8 ffffffff8108007d
>>  ffffea0000141720 000001ff41334434 ffff880000000001 ffff88004d3aedc0
>> Call Trace:
>>  [<ffffffff8136b76f>] dump_stack+0x67/0x98
>>  [<ffffffff8108007d>] __warn+0xfd/0x120
>>  [<ffffffff810800bd>] warn_slowpath_null+0x1d/0x20
>>  [<ffffffff814ec0a8>] xen_blkbk_remove+0x138/0x140
>>  [<ffffffff81449b07>] xenbus_dev_remove+0x47/0xa0
>>  [<ffffffff814bd2b4>] __device_release_driver+0xb4/0x160
>>  [<ffffffff814bd38d>] device_release_driver+0x2d/0x40
>>  [<ffffffff814bc2b4>] bus_remove_device+0x124/0x190
>>  [<ffffffff814b9682>] device_del+0x112/0x210
>>  [<ffffffff81448423>] ? xenbus_read+0x53/0x70
>>  [<ffffffff814b97a2>] device_unregister+0x22/0x60
>>  [<ffffffff814eda9d>] frontend_changed+0xad/0x4c0
>>  [<ffffffff81449e67>] xenbus_otherend_changed+0xc7/0x140
>>  [<ffffffff816f1486>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>  [<ffffffff8144a2f0>] frontend_changed+0x10/0x20
>>  [<ffffffff81447b0c>] xenwatch_thread+0x9c/0x140
>>  [<ffffffff810bffb0>] ? woken_wake_function+0x20/0x20
>>  [<ffffffff816ed98a>] ? schedule+0x3a/0xa0
>>  [<ffffffff816f1486>] ? _raw_spin_unlock_irqrestore+0x16/0x20
>>  [<ffffffff810c0c6d>] ? complete+0x4d/0x60
>>  [<ffffffff81447a70>] ? split+0xf0/0xf0
>>  [<ffffffff810a0535>] kthread+0xe5/0x100
>>  [<ffffffff810a051d>] ? kthread+0xcd/0x100
>>  [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>  [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>  [<ffffffff810a0450>] ? __kthread_init_worker+0x40/0x40
>>  [<ffffffff816f1bc5>] ret_from_fork+0x25/0x30
>> ---[ end trace ea3a48c80e4ad79d ]---
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@xxxxxxxxxxxxx
>> https://lists.xen.org/xen-devel
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel