Re: [Xen-users] 2.6.31 xenified kernel - not ready for production

[Pasi Kärkkäinen <pasik@xxxxxx>]

On Mon, Nov 09, 2009 at 06:20:01PM +0000, Andrew Lyon wrote:
> On Sun, Nov 8, 2009 at 8:32 PM, Pasi Kärkkäinen <pasik@xxxxxx> wrote:
> > On Sun, Nov 08, 2009 at 08:40:43PM +0100, Peter Braun wrote:
> >> Hi,
> >>
> >> I just want to know if somebody use 2.6.31.4 xenified kernel (aka
> >> OpenSUSE) in production?
> >>
> >> We have been testing it on new Nehalem Xeon server for few weeks w/o
> >> any problem.
> >> But as soon we tried it on production machine - after several
> >> production domUs started - hard OS failure.
> >> We had to switch back to 2.6.18.8 - xen stock kernel.
> >>
> >
> > What kind of failure?
> >
> > I hope you have a serial console set up so you can capture the (possible)
> > BUG/OOPS/stacktrace or just the error messages..
> >
> > -- Pasi
> >
> >
> 
> Yes I've had problems with 2.6.31 crashing with null pointer
> dereference, I am going to install opensuse 11.2 next week and see if
> I can replicate the problem using the opensuse kernel so that I can
> get some help from Jan to fix it.
> 

Hmm.. dunno if this helps, it's from a user (from #xen) having problems with the
opensuse 2.6.31 patches:

http://hachi.kuiki.net/software_problems/20091110-xen-i7-panic/panic2.txt

[  257.758664] BUG: unable to handle kernel paging request at ffff88016f521000
[  257.766783] IP: [<ffffffff80391feb>] swiotlb_bounce+0x35/0x3a
[  257.773349] PGD 1fd4067 PUD 29da067 PMD 2b55067 PTE 0
[  257.779308] Thread overran stack, or stack corrupted
[  257.784979] Oops: 0002 [#1] SMP 
[  257.789046] last sysfs file: 
/sys/devices/xen-backend/vbd-1-2059/statistics/wr_sect
[  257.797632] CPU 0 
[  257.800027] Modules linked in: xt_tcpudp xt_physdev iptable_filter ip_tables 
x_tables bridge nls_utf8 nls_cp437 vfat fat 8021q garp stp bonding ipv6 lm85 
hwmon_vid i2c_amd756 i2c_i801 i2c_core pl2303 usbserial button pcspkr processor 
evdev ext3 jbd dm_mod raid456 raid6_pq async_xor async_memcpy async_tx xor 
raid1 raid0 md_mod sd_mod ata_generic sata_promise ata_piix libata uhci_hcd 
scsi_mod ide_pci_generic ide_core ehci_hcd e1000e thermal fan thermal_sys 
configfs e100 mii [last unloaded: scsi_wait_scan]
[  257.853871] Pid: 0, comm: swapper Not tainted 2.6.31.5 #2 X8ST3
[  257.860462] RIP: e030:[<ffffffff80391feb>]  [<ffffffff80391feb>] 
swiotlb_bounce+0x35/0x3a
[  257.869634] RSP: e02b:ffffc90000003d38  EFLAGS: 00010002
[  257.875562] RAX: 0000000000002000 RBX: 0000000000007748 RCX: 0000000000002000
[  257.883513] RDX: 0000000000002000 RSI: ffff88000e823000 RDI: ffff88016f521000
[  257.891459] RBP: 0000000000000002 R08: ffff88000e823000 R09: 000000016f521000
[  257.899379] R10: 000020208065dc20 R11: ffffc90000003e50 R12: 0000000000007748
[  257.907331] R13: 0000000000002000 R14: ffff8801718bd080 R15: 0000000000000050
[  257.915284] FS:  00007fb30d7106f0(0000) GS:ffffc90000000000(0000) 
knlGS:0000000000000000
[  257.924274] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[  257.930694] CR2: ffff88016f521000 CR3: 0000000170aa8000 CR4: 0000000000002660
[  257.938606] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  257.946541] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  257.954466] Process swapper (pid: 0, threadinfo ffffffff805de000, task 
ffffffff8062b490)
[  257.963447] Stack:
[  257.965740]  ffffffff802b0358 ffffffff80392066 ffffffff802b0358 
ffff8800f10220c0
[  257.974011] <0> 0000000000000001 0000000000000002 0000000000000002 
ffffffff80392197
[  257.982945] <0> ffff88016f2f80d0 ffff88016f2f8000 0000000000000002 
ffff8800f10220c0
[  257.997334] Call Trace:
[  258.000107]  <IRQ> 
[  258.002580]  [<ffffffff802b0358>] ? kfree+0x68/0x1a0
[  258.008121]  [<ffffffff80392066>] ? unmap_single+0x76/0x172
[  258.014315]  [<ffffffff802b0358>] ? kfree+0x68/0x1a0
[  258.019857]  [<ffffffff80392197>] ? swiotlb_unmap_sg_attrs+0x35/0x4e
[  258.026939]  [<ffffffffa00d3253>] ? ata_sg_clean+0x8a/0xa2 [libata]
[  258.033938]  [<ffffffffa00d32dd>] ? __ata_qc_complete+0x72/0xe1 [libata]
[  258.041406]  [<ffffffffa00e286f>] ? ata_sff_hsm_move+0x660/0x68b [libata]
[  258.048956]  [<ffffffffa004c7c1>] ? e1000_clean_tx_irq+0xc5/0x2d9 [e1000e]
[  258.056599]  [<ffffffffa00e2977>] ? ata_sff_host_intr+0xdd/0x128 [libata]
[  258.064161]  [<ffffffffa00e2af4>] ? ata_sff_interrupt+0x85/0xbf [libata]
[  258.071609]  [<ffffffff8026dc17>] ? handle_IRQ_event+0x74/0x147
[  258.078197]  [<ffffffff80238932>] ? __do_softirq+0x171/0x1b8
[  258.084498]  [<ffffffff8026f4d8>] ? handle_level_irq+0x9e/0x104
[  258.091081]  [<ffffffff8020b601>] ? handle_irq+0x17/0x1d
[  258.097052]  [<ffffffff804087f8>] ? evtchn_do_upcall+0x12d/0x1fd
[  258.103741]  [<ffffffff80209a4e>] ? do_hypervisor_callback+0x1e/0x30
[  258.110808]  <EOI> 
[  258.113281]  [<ffffffff8020bdef>] ? xen_safe_halt+0xa2/0xb7
[  258.119483]  [<ffffffff8020f2c5>] ? xen_idle+0x5e/0xbc
[  258.125213]  [<ffffffff802087fe>] ? cpu_idle+0x46/0x82
[  258.130932]  [<ffffffff8068324b>] ? start_kernel+0x37b/0x387
[  258.137262] Code: 89 f0 48 89 d0 75 13 48 be 00 00 00 00 00 88 ff ff 48 8d 
34 37 4c 89 c7 eb 0e 48 bf 00 00 00 00 00 88 ff ff 49 8d 3c 39 48 89 c1 <f3> a4 
41 58 c3 41 55 49 89 d5 41 54 55 89 cd 53 48 89 f3 48 83 
[  258.163334] RIP  [<ffffffff80391feb>] swiotlb_bounce+0x35/0x3a
[  258.169904]  RSP <ffffc90000003d38>
[  258.173813] CR2: ffff88016f521000
[  258.177542] ---[ end trace f10a55534d9fba8d ]---
[  258.182794] Kernel panic - not syncing: Fatal exception in interrupt
[  258.189843] Pid: 0, comm: swapper Tainted: G      D    2.6.31.5 #2
[  258.196716] Call Trace:
[  258.199489]  <IRQ>  [<ffffffff804d7710>] ? panic+0x86/0x14c
[  258.205811]  [<ffffffff8024afe0>] ? up+0xe/0x36
[  258.210876]  [<ffffffff802335d6>] ? release_console_sem+0x1e6/0x21b
[  258.217839]  [<ffffffff8020d251>] ? oops_end+0xbe/0xcb
[  258.223571]  [<ffffffff80217480>] ? no_context+0x1fc/0x20b
[  258.229693]  [<ffffffffa030596b>] ? br_handle_frame_finish+0x127/0x148 
[bridge]
[  258.237833]  [<ffffffff8021763f>] ? __bad_area_nosemaphore+0x1b0/0x1d4
[  258.245128]  [<ffffffffa0309a50>] ? br_nf_pre_routing_finish+0x0/0x2d2 
[bridge]
[  258.253257]  [<ffffffff8047b3f0>] ? nf_hook_slow+0x62/0xc3
[  258.259377]  [<ffffffffa0309a50>] ? br_nf_pre_routing_finish+0x0/0x2d2 
[bridge]
[  258.267512]  [<ffffffffa0304e93>] ? __br_forward+0x88/0x9d [bridge]
[  258.274506]  [<ffffffff8021786c>] ? do_page_fault+0xa2/0x27a
[  258.280810]  [<ffffffff804da4f8>] ? page_fault+0x28/0x30
[  258.286727]  [<ffffffff80391feb>] ? swiotlb_bounce+0x35/0x3a
[  258.293059]  [<ffffffff802b0358>] ? kfree+0x68/0x1a0
[  258.298637]  [<ffffffff80392066>] ? unmap_single+0x76/0x172
[  258.304856]  [<ffffffff802b0358>] ? kfree+0x68/0x1a0
[  258.310408]  [<ffffffff80392197>] ? swiotlb_unmap_sg_attrs+0x35/0x4e
[  258.317472]  [<ffffffffa00d3253>] ? ata_sg_clean+0x8a/0xa2 [libata]
[  258.324456]  [<ffffffffa00d32dd>] ? __ata_qc_complete+0x72/0xe1 [libata]
[  258.331948]  [<ffffffffa00e286f>] ? ata_sff_hsm_move+0x660/0x68b [libata]
[  258.339493]  [<ffffffffa004c7c1>] ? e1000_clean_tx_irq+0xc5/0x2d9 [e1000e]
[  258.347171]  [<ffffffffa00e2977>] ? ata_sff_host_intr+0xdd/0x128 [libata]
[  258.354732]  [<ffffffffa00e2af4>] ? ata_sff_interrupt+0x85/0xbf [libata]
[  258.362187]  [<ffffffff8026dc17>] ? handle_IRQ_event+0x74/0x147
[  258.368778]  [<ffffffff80238932>] ? __do_softirq+0x171/0x1b8
[  258.375069]  [<ffffffff8026f4d8>] ? handle_level_irq+0x9e/0x104
[  258.381658]  [<ffffffff8020b601>] ? handle_irq+0x17/0x1d
[  258.387582]  [<ffffffff804087f8>] ? evtchn_do_upcall+0x12d/0x1fd
[  258.394268]  [<ffffffff80209a4e>] ? do_hypervisor_callback+0x1e/0x30
[  258.401358]  <EOI>  [<ffffffff8020bdef>] ? xen_safe_halt+0xa2/0xb7
[  258.408316]  [<ffffffff8020f2c5>] ? xen_idle+0x5e/0xbc
[  258.414054]  [<ffffffff802087fe>] ? cpu_idle+0x46/0x82
[  258.419816]  [<ffffffff8068324b>] ? start_kernel+0x37b/0x387
[  258.426378] Rebooting in 60 seconds..  
s <SpaceBar> to update BIOS.


-- Pasi


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users