[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] 100% reliable Oops on xen 4.0.1



On Mon, Aug 13, 2012 at 05:03:06PM -0700, Peter Moody wrote:
> This seems to be some combination of Xen and the audit subsystem, but
> the attached program crashes my machine 100% of the time.
> 

Did you try with a later Xen version? 4.0.1 is quite old. 
For example the latest in Xen 4.0.x series which is 4.0.4 ? Or Xen 4.1.3 ? 

-- Pasi

> steps to reproduce the crash:
> 
>  *  1) compile with gcc -m32
>  *  2) start auditd, install any rule (I've only tested syscall
> auditing, but any syscall seems to work).
>  *     /etc/init.d/auditd start ; auditctl -D ; auditctl -a
> exit,always -F arch=64 -S chmod
>  *  3) run'n wait (this only loops twice for me before dying)
>  *     ./a.out
>  *  4) bask in instantaneous kernel oops.
> 
> here's xm info from dom0
> 
> [xen2.atl] root@gntb1:~# xm info
> host                   : gntb1.atl.corp.google.com
> release                : 3.2.13-ganeti-rx6-xen0
> version                : #1 SMP Thu Jun 7 12:59:40 CEST 2012
> machine                : x86_64
> nr_cpus                : 12
> nr_nodes               : 2
> cores_per_socket       : 6
> threads_per_core       : 1
> cpu_mhz                : 2660
> hw_caps                :
> bfebfbff:2c100800:00000000:00001f40:029ee3ff:00000000:00000001:00000000
> virt_caps              : hvm
> total_memory           : 32755
> free_memory            : 22665
> node_to_cpu            : node0:0,2,4,6,8,10
>                          node1:1,3,5,7,9,11
> node_to_memory         : node0:13083
>                          node1:9582
> node_to_dma32_mem      : node0:0
>                          node1:3235
> max_node_id            : 1
> xen_major              : 4
> xen_minor              : 0
> xen_extra              : .1
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          : unavailable
> xen_commandline        : placeholder dom0_mem=1024M loglvl=all
> com1=115200,8n1 console=com1 iommu=0
> cc_compiler            : gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5)
> cc_compile_by          : pmacedo
> cc_compile_domain      : google.com
> cc_compile_date        : Wed Mar 16 15:24:06 UTC 2011
> xend_config_format     : 4
> 
> I'm not sure what you need from the domU. It's running 2.6.38.8 (but
> I've seen this bug all the way up to 3.5.0-rc7, the latest I've
> tested). It's a fairly beefy setup, 32G memory and 6 cpus.
> 
> I suspect xen as opposed to auditd because:
> 
>  a) this only happens on our xen machines (though not all of them)
>  b) one of my stack traces started with
> 
> [172577.560441]  [<ffffffff810065ad>] ? xen_force_evtchn_callback+0xd/0x10
> 
> Any one have any idea what's going on?
> 
> Cheers,
> peter
> 
> -- 
> Peter Moody      Google    1.650.253.7306
> Security Engineer  pgp:0xC3410038

> /*
>  * steps:
>  *  1) compile with gcc -m32
>  *  2) start auditd, install any rule (I've only tested syscall auditing, but 
> any syscall seems to work).
>  *     /etc/init.d/auditd start ; auditctl -D ; auditctl -a exit,always -F 
> arch=64 -S chmod
>  *  3) run'n wait (this only loops twice for me before dying)
>  *     ./a.out
>  *  4) bask in instantaneous kernel oops.
>  [  571.282777] ------------[ cut here ]------------
>  [  571.282786] kernel BUG at fs/buffer.c:1263!
>  [  571.282790] invalid opcode: 0000 [#1] SMP
>  [  571.282795] last sysfs file: 
> /sys/devices/system/cpu/sched_mc_power_savings
>  [  571.282798] CPU 0
>  [  571.282802] Pid: 7457, comm: a.out Not tainted 2.6.38.8-gg868-ganetixenu 
> #1
>  [  571.282808] RIP: e030:[<ffffffff81153853>]  [<ffffffff81153853>] 
> __find_get_block+0x1f3/0x200
>  [  571.282819] RSP: e02b:ffff88079b7ddc78  EFLAGS: 00010046
>  [  571.282822] RAX: ffff8807bc290000 RBX: ffff8806d9bb9a98 RCX: 
> 00000000023dc17c
>  [  571.282826] RDX: 0000000000001000 RSI: 00000000023dc17c RDI: 
> ffff8807fec29a00
>  [  571.282830] RBP: ffff88079b7ddcd8 R08: 0000000000000001 R09: 
> ffff8806d9bb99c0
>  [  571.282834] R10: 0000000000000000 R11: 0000000000000000 R12: 
> ffff8806d9bb99c4
>  [  571.282839] R13: ffff8806d9bb99f0 R14: ffff8807feff9060 R15: 
> 00000000023dc17c
>  [  571.282845] FS:  00007f8f6a76a7c0(0000) GS:ffff8807fff26000(0063) 
> knlGS:0000000000000000
>  [  571.282849] CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
>  [  571.282853] CR2: 00000000f76c6970 CR3: 00000007a250b000 CR4: 
> 0000000000002660
>  [  571.282857] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
> 0000000000000000
>  [  571.282861] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
> 0000000000000400
>  [  571.282866] Process a.out (pid: 7457, threadinfo ffff88079b7dc000, task 
> ffff8807786843e0)
>  [  571.282870] Stack:
>  [  571.282872]  ffff88079b7ddc98 ffffffff81654cd1 ffff88079b7ddca8 
> ffff8806d9bba440
>  [  571.282879]  ffff88079b7ddd08 ffffffff811c9294 ffff8807ffffffc3 
> 0000000000000014
>  [  571.282887]  ffff8806d9bb9a98 ffff8806d9bb99c4 ffff8806d9bb99f0 
> ffff8807feff9060
>  [  571.282895] Call Trace:
>  [  571.282901]  [<ffffffff81654cd1>] ? down_read+0x11/0x30
>  [  571.282907]  [<ffffffff811c9294>] ? ext3_xattr_get+0xf4/0x2b0
>  [  571.282913]  [<ffffffff811baf88>] ext3_clear_blocks+0x128/0x190
>  [  571.282918]  [<ffffffff811bb104>] ext3_free_data+0x114/0x160
>  [  571.282923]  [<ffffffff811bbc0a>] ext3_truncate+0x87a/0x950
>  [  571.282928]  [<ffffffff812133f5>] ? journal_start+0xb5/0x100
>  [  571.282933]  [<ffffffff811bc840>] ext3_evict_inode+0x180/0x1a0
>  [  571.282938]  [<ffffffff8114065f>] evict+0x1f/0xb0
>  [  571.282945]  [<ffffffff81006d52>] ? check_events+0x12/0x20
>  [  571.282949]  [<ffffffff81140c14>] iput+0x1a4/0x290
>  [  571.282955]  [<ffffffff8113ed05>] dput+0x265/0x310
>  [  571.282959]  [<ffffffff81132435>] path_put+0x15/0x30
>  [  571.282965]  [<ffffffff810a5d31>] audit_syscall_exit+0x171/0x260
>  [  571.282971]  [<ffffffff8103ed9a>] sysexit_audit+0x21/0x5f
>  [  571.282974] Code: 82 00 05 01 00 85 c0 75 de 65 48 89 1c 25 00 05 01 00 
> e9 87 fe ff ff 48 89 df e8 e9 fc ff ff 4c 89 f7 e9 02 ff ff ff 0f 0b eb fe 
> <0f> 0b eb fe 0f 0b eb fe 0f 1f 44 00 00 55 48 89 e5 41 57 49 89
>  [  571.283027] RIP  [<ffffffff81153853>] __find_get_block+0x1f3/0x200
>  [  571.283033]  RSP <ffff88079b7ddc78>
>  [  571.283036] ---[ end trace 5975ffe20808ecd2 ]---
>  *
>  */
> 
> #include <stdio.h>
> #include <sys/stat.h>
> #include <sys/types.h>
> #include <unistd.h>
> 
> #define KILLDIR "/usr/local/tmp/crasher/kill_dir"
> 
> int main(void) {
>   FILE *f;
>   char fullpath[512];
>   int i;
> 
>   while (1) {
>     fprintf(stderr, "%d ", i++);
>     mkdir(KILLDIR, 0777);
>     chdir(KILLDIR);
>     sprintf(fullpath, "%s/file", KILLDIR);
>     f = fopen(fullpath, "w+");
>     fprintf(f, "nothing to see here");
>     fclose(f);
>     unlink("/usr/local/tmp/crasher/kill_dir/file");
>     rmdir(KILLDIR);
>   }
>   return 0;
> }

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.