[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] Re: Xen question



On 02/01/2011 10:49 AM, Stephen Hemminger wrote:
> Something is wrong with the Vyatta Xen kernel.
> It worked with 2.6.32, but something wrong with 2.6.35.
> Have you seen this?
>
> --
> [    0.000000] Reserving virtual address space above 0xf5800000
> [    0.000000] Initializing cgroup subsys cpuset
> [    0.000000] Initializing cgroup subsys cpu
> [    0.000000] Linux version 2.6.35-1-586-vyatta-virt (root@vyatta) (gcc 
> version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Jan 28 16:40:27 PST 2011
> [    0.000000] ACPI in unprivileged domain disabled
> [    0.000000] BIOS-provided physical RAM map:
> [    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
> [    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
> [    0.000000]  Xen: 0000000000100000 - 0000000020000000 (usable)
> [    0.000000] NX (Execute Disable) protection: active
> [    0.000000] DMI not present or invalid.
> [    0.000000] last_pfn = 0x20000 max_arch_pfn = 0x1000000
> [    0.000000] init_memory_mapping: 0000000000000000-0000000020000000
> [    0.000000] RAMDISK: 01607000 - 03148000
> [    0.000000] Node: 0, start_pfn: 0, end_pfn: 20000
> [    0.000000] Reserving total of 600 pages for numa KVA remap
> [    0.000000] kva_start_pfn ~ 1f400 max_low_pfn ~ 20000
> [    0.000000] max_pfn = 20000
> [    0.000000] 0MB HIGHMEM available.
> [    0.000000] 512MB LOWMEM available.
> [    0.000000] 1 multicall(s) failed: cpu 0
> [    0.000000] Pid: 0, comm: swapper Not tainted 2.6.35-1-586-vyatta-virt #1
> [    0.000000] Call Trace:
> [    0.000000]  [<c100415b>] ? xen_mc_flush+0x92/0x13d
> [    0.000000]  [<c1004a39>] ? xen_mc_issue+0x11/0x1f
> [    0.000000]  [<c10053f9>] ? xen_set_pmd_hyper+0x55/0x5c
> [    0.000000]  [<c1025d25>] ? set_pmd_pfn+0xb4/0xc3
> [    0.000000]  [<c13df48f>] ? initmem_init+0x4e9/0x5ea
> [    0.000000]  [<c12895ca>] ? printk+0xe/0x14
> [    0.000000]  [<c13d151c>] ? setup_arch+0x8d1/0x989
> [    0.000000]  [<c13e0020>] ? proc_caches_init+0x27/0xbf
> [    0.000000]  [<c13cd61e>] ? start_kernel+0x86/0x330
> [    0.000000]  [<c13cfabf>] ? xen_start_kernel+0x4c0/0x4c7

This is failing during the initial mapping of memory, I think.  Are
there any messages on the Xen console?  Is it completely deterministic?

> [    0.000000] ------------[ cut here ]------------
> [    0.000000] WARNING: at arch/x86/xen/multicalls.c:182 
> xen_mc_issue+0x11/0x1f()
> [    0.000000] Modules linked in:
> [    0.000000] Pid: 0, comm: swapper Not tainted 2.6.35-1-586-vyatta-virt #1
> [    0.000000] Call Trace:
> [    0.000000]  [<c10357ce>] ? warn_slowpath_common+0x6a/0x7b
> [    0.000000]  [<c1004a39>] ? xen_mc_issue+0x11/0x1f
> [    0.000000]  [<c10357ec>] ? warn_slowpath_null+0xd/0x10
> [    0.000000]  [<c1004a39>] ? xen_mc_issue+0x11/0x1f
> [    0.000000]  [<c10053f9>] ? xen_set_pmd_hyper+0x55/0x5c
> [    0.000000]  [<c1025d25>] ? set_pmd_pfn+0xb4/0xc3
> [    0.000000]  [<c13df48f>] ? initmem_init+0x4e9/0x5ea
> [    0.000000]  [<c12895ca>] ? printk+0xe/0x14
> [    0.000000]  [<c13d151c>] ? setup_arch+0x8d1/0x989
> [    0.000000]  [<c13e0020>] ? proc_caches_init+0x27/0xbf
> [    0.000000]  [<c13cd61e>] ? start_kernel+0x86/0x330
> [    0.000000]  [<c13cfabf>] ? xen_start_kernel+0x4c0/0x4c7
> [    0.000000] ---[ end trace 4eaa2a86a8e2da22 ]---
> [    0.000000] 1 multicall(s) failed: cpu 0
> [    0.000000] Pid: 0, comm: swapper Tainted: G        W   
> 2.6.35-1-586-vyatta-virt #1
> [    0.000000] Call Trace:
> [    0.000000]  [<c100415b>] ? xen_mc_flush+0x92/0x13d
> [    0.000000]  [<c1004a39>] ? xen_mc_issue+0x11/0x1f
> [    0.000000]  [<c10053f9>] ? xen_set_pmd_hyper+0x55/0x5c
> [    0.000000]  [<c1025d25>] ? set_pmd_pfn+0xb4/0xc3
> [    0.000000]  [<c13df48f>] ? initmem_init+0x4e9/0x5ea
> [    0.000000]  [<c12895ca>] ? printk+0xe/0x14
> [    0.000000]  [<c13d151c>] ? setup_arch+0x8d1/0x989
> [    0.000000]  [<c13e0020>] ? proc_caches_init+0x27/0xbf
> [    0.000000]  [<c13cd61e>] ? start_kernel+0x86/0x330
> [    0.000000]  [<c13cfabf>] ? xen_start_kernel+0x4c0/0x4c7
> [    0.000000] ------------[ cut here ]------------
> [    0.000000] WARNING: at arch/x86/xen/multicalls.c:182 
> xen_mc_issue+0x11/0x1f()
> [    0.000000] Modules linked in:
> [    0.000000] Pid: 0, comm: swapper Tainted: G        W   
> 2.6.35-1-586-vyatta-virt #1
> [    0.000000] Call Trace:
> [    0.000000]  [<c10357ce>] ? warn_slowpath_common+0x6a/0x7b
> [    0.000000]  [<c1004a39>] ? xen_mc_issue+0x11/0x1f
> [    0.000000]  [<c10357ec>] ? warn_slowpath_null+0xd/0x10
> [    0.000000]  [<c1004a39>] ? xen_mc_issue+0x11/0x1f
> [    0.000000]  [<c10053f9>] ? xen_set_pmd_hyper+0x55/0x5c
> [    0.000000]  [<c1025d25>] ? set_pmd_pfn+0xb4/0xc3
> [    0.000000]  [<c13df48f>] ? initmem_init+0x4e9/0x5ea
> [    0.000000]  [<c12895ca>] ? printk+0xe/0x14
> [    0.000000]  [<c13d151c>] ? setup_arch+0x8d1/0x989
> [    0.000000]  [<c13e0020>] ? proc_caches_init+0x27/0xbf
> [    0.000000]  [<c13cd61e>] ? start_kernel+0x86/0x330
> [    0.000000]  [<c13cfabf>] ? xen_start_kernel+0x4c0/0x4c7
> [    0.000000] ---[ end trace 4eaa2a86a8e2da23 ]---
> [    0.000000] 1 multicall(s) failed: cpu 0
> [    0.000000] Pid: 0, comm: swapper Tainted: G        W   
> 2.6.35-1-586-vyatta-virt #1
> [    0.000000] Call Trace:
> [    0.000000]  [<c100415b>] ? xen_mc_flush+0x92/0x13d
> [    0.000000]  [<c1004a39>] ? xen_mc_issue+0x11/0x1f
> [    0.000000]  [<c10053f9>] ? xen_set_pmd_hyper+0x55/0x5c
> [    0.000000]  [<c1025d25>] ? set_pmd_pfn+0xb4/0xc3
> [    0.000000]  [<c13df48f>] ? initmem_init+0x4e9/0x5ea
> [    0.000000]  [<c12895ca>] ? printk+0xe/0x14
> [    0.000000]  [<c13d151c>] ? setup_arch+0x8d1/0x989
> [    0.000000]  [<c13e0020>] ? proc_caches_init+0x27/0xbf
> [    0.000000]  [<c13cd61e>] ? start_kernel+0x86/0x330
> [    0.000000]  [<c13cfabf>] ? xen_start_kernel+0x4c0/0x4c7
> [    0.000000] ------------[ cut here ]------------
> [    0.000000] WARNING: at arch/x86/xen/multicalls.c:182 
> xen_mc_issue+0x11/0x1f()
> [    0.000000] Modules linked in:
> [    0.000000] Pid: 0, comm: swapper Tainted: G        W   
> 2.6.35-1-586-vyatta-virt #1
> [    0.000000] Call Trace:
> [    0.000000]  [<c10357ce>] ? warn_slowpath_common+0x6a/0x7b
> [    0.000000]  [<c1004a39>] ? xen_mc_issue+0x11/0x1f
> [    0.000000]  [<c10357ec>] ? warn_slowpath_null+0xd/0x10
> [    0.000000]  [<c1004a39>] ? xen_mc_issue+0x11/0x1f
> [    0.000000]  [<c10053f9>] ? xen_set_pmd_hyper+0x55/0x5c
> [    0.000000]  [<c1025d25>] ? set_pmd_pfn+0xb4/0xc3
> [    0.000000]  [<c13df48f>] ? initmem_init+0x4e9/0x5ea
> [    0.000000]  [<c12895ca>] ? printk+0xe/0x14
> [    0.000000]  [<c13d151c>] ? setup_arch+0x8d1/0x989
> [    0.000000]  [<c13e0020>] ? proc_caches_init+0x27/0xbf
> [    0.000000]  [<c13cd61e>] ? start_kernel+0x86/0x330
> [    0.000000]  [<c13cfabf>] ? xen_start_kernel+0x4c0/0x4c7
> [    0.000000] ---[ end trace 4eaa2a86a8e2da24 ]---
> [    0.000000]   mapped low ram: 0 - 20000000
> [    0.000000]   low ram: 0 - 20000000
> [    0.000000] Zone PFN ranges:
> [    0.000000]   DMA      0x00000001 -> 0x00001000
> [    0.000000]   Normal   0x00001000 -> 0x00020000
> [    0.000000]   HighMem  empty
> [    0.000000] Movable zone start PFN for each node
> [    0.000000] early_node_map[2] active PFN ranges
> [    0.000000]     0: 0x00000001 -> 0x000000a0
> [    0.000000]     0: 0x00000100 -> 0x0001fa00
> [    0.000000] Using APIC driver default
> [    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org
> [    0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs
> [    0.000000] Local APIC disabled by BIOS -- you can enable it with "lapic"
> [    0.000000] APIC: disable apic facility
> [    0.000000] APIC: switched to apic NOOP
> [    0.000000] PM: Registered nosave memory: 00000000000a0000 - 
> 0000000000100000
> [    0.000000] Allocating PCI resources starting at 20000000 (gap: 
> 20000000:e0000000)
> [    0.000000] Booting paravirtualized kernel on Xen
> [    0.000000] Xen version: 4.0.1 (preserve-AD)
> [    0.000000] setup_percpu: NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:1 
> nr_node_ids:8
> [    0.000000] PERCPU: Embedded 15 pages/cpu @c14e7000 s37504 r0 d23936 u65536
> [    0.000000] pcpu-alloc: s37504 r0 d23936 u65536 alloc=16*4096
> [    0.000000] pcpu-alloc: [0] 0
> [    0.000000] Xen: using vcpu_info placement
> [    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total 
> pages: 128427
> [    0.000000] Policy zone: Normal
> [    0.000000] Kernel command line: root=/dev/xvda1 ro 
> ip=:127.0.255.255::::eth0:dhcp
> [    0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
> [    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
> [    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
> [    0.000000] Enabling fast FPU save and restore... done.
> [    0.000000] Enabling unmasked SIMD FPU exception support... done.
> [    0.000000] Initializing CPU#0
> [    0.000000] Subtract (33 early reservations)
> [    0.000000]   #1 [00031cb000 - 00031e7000]  XEN PAGETABLES
> [    0.000000]   #2 [0000001000 - 0000002000]   EX TRAMPOLINE
> [    0.000000]   #3 [0001000000 - 00014e15d0]   TEXT DATA BSS
> [    0.000000]   #4 [0001607000 - 0003148000]         RAMDISK
> [    0.000000]   #5 [0003148000 - 00031cb000]  XEN START INFO
> [    0.000000]   #6 [0000002000 - 0000003000]      TRAMPOLINE
> [    0.000000]   #7 [0000003000 - 0000007000]     ACPI WAKEUP
> [    0.000000]   #8 [0000100000 - 00001e4000]         PGTABLE
> [    0.000000]   #9 [001fa00000 - 0020000000]         KVA RAM
> [    0.000000]   #10 [001f400000 - 001fa00000]          KVA PG
> [    0.000000]   #11 [00014e2000 - 00014e3000]         BOOTMEM
> [    0.000000]   #12 [00014e1600 - 00014e1604]         BOOTMEM
> [    0.000000]   #13 [00014e1640 - 00014e1700]         BOOTMEM
> [    0.000000]   #14 [00014e1700 - 00014e1730]         BOOTMEM
> [    0.000000]   #15 [00014e3000 - 00014e4800]         BOOTMEM
> [    0.000000]   #16 [00014e1740 - 00014e17d0]         BOOTMEM
> [    0.000000]   #17 [00014e1800 - 00014e1840]         BOOTMEM
> [    0.000000]   #18 [00014e1840 - 00014e1880]         BOOTMEM
> [    0.000000]   #19 [00014e1880 - 00014e18c0]         BOOTMEM
> [    0.000000]   #20 [00014e18c0 - 00014e18d0]         BOOTMEM
> [    0.000000]   #21 [00014e1900 - 00014e1933]         BOOTMEM
> [    0.000000]   #22 [00014e1940 - 00014e1973]         BOOTMEM
> [    0.000000]   #23 [00014e7000 - 00014f6000]         BOOTMEM
> [    0.000000]   #24 [00014e1980 - 00014e1984]         BOOTMEM
> [    0.000000]   #25 [00014e19c0 - 00014e19c4]         BOOTMEM
> [    0.000000]   #26 [00014e1a00 - 00014e1a04]         BOOTMEM
> [    0.000000]   #27 [00014e1a40 - 00014e1a44]         BOOTMEM
> [    0.000000]   #28 [00014e1a80 - 00014e1b00]         BOOTMEM
> [    0.000000]   #29 [00014e1b00 - 00014e1b2c]         BOOTMEM
> [    0.000000]   #30 [00014e4800 - 00014e6800]         BOOTMEM
> [    0.000000]   #31 [00014f6000 - 0001536000]         BOOTMEM
> [    0.000000]   #32 [0001536000 - 0001556000]         BOOTMEM
> [    0.000000] Initializing HighMem for node 0 (00000000:00000000)
> [    0.000000] Memory: 476668k/524288k available (2608k kernel code, 41088k 
> reserved, 1281k data, 512k init, 0k highmem)
> [    0.000000] virtual kernel memory layout:
> [    0.000000]     fixmap  : 0xf5536000 - 0xf57ff000   (2852 kB)
> [    0.000000]     pkmap   : 0xf5200000 - 0xf5400000   (2048 kB)
> [    0.000000]     vmalloc : 0xe0800000 - 0xf51fe000   ( 329 MB)
> [    0.000000]     lowmem  : 0xc0000000 - 0xe0000000   ( 512 MB)
> [    0.000000]       .init : 0xc13cd000 - 0xc144d000   ( 512 kB)
> [    0.000000]       .data : 0xc128c1b2 - 0xc13cc918   (1281 kB)
> [    0.000000]       .text : 0xc1000000 - 0xc128c1b2   (2608 kB)
> [    0.000000] SLUB: Genslabs=14, HWalign=64, Order=0-3, MinObjects=0, 
> CPUs=1, Nodes=8
> [    0.000000] Hierarchical RCU implementation.
> [    0.000000]  RCU dyntick-idle grace-period acceleration is enabled.
> [    0.000000]  RCU-based detection of stalled CPUs is disabled.
> [    0.000000]  Verbose stalled-CPUs detection is disabled.
> [    0.000000] NR_IRQS:2304 nr_irqs:256
> [    0.000000] Console: colour dummy device 80x25
> [    0.000000] console [tty0] enabled
> [    0.000000] console [hvc0] enabled
> [    0.000000] installing Xen timer for CPU 0
> [    0.000000] Detected 1994.378 MHz processor.
> [    0.010000] Calibrating delay loop (skipped), value calculated using timer 
> frequency.. 3988.75 BogoMIPS (lpj=19943780)
> [    0.010000] pid_max: default: 32768 minimum: 301
> [    0.010000] Security Framework initialized
> [    0.010000] SELinux:  Disabled at boot.
> [    0.010000] Mount-cache hash table entries: 512
> [    0.010000] Initializing cgroup subsys ns
> [    0.010000] Initializing cgroup subsys cpuacct
> [    0.010000] Initializing cgroup subsys devices
> [    0.010000] Initializing cgroup subsys freezer
> [    0.010000] Performance Events:
> [    0.010000] no APIC, boot with the "lapic" boot parameter to force-enable 
> it.
> [    0.010000] no hardware sampling interrupt available.
> [    0.010000] AMD PMU driver.
> [    0.010000] ... version:                0
> [    0.010000] ... bit width:              48
> [    0.010000] ... generic registers:      4
> [    0.010000] ... value mask:             0000ffffffffffff
> [    0.010000] ... max period:             00007fffffffffff
> [    0.010000] ... fixed-purpose events:   0
> [    0.010000] ... event mask:             000000000000000f
> [    0.010000] SMP alternatives: switching to UP code
> [    0.010000] Freeing SMP alternatives: 16k freed
> [    0.010121] cpu 0 spinlock event irq 1
> [    0.010429] Brought up 1 CPUs
> [    0.010467] BUG: unable to handle kernel NULL pointer dereference at (null)
> [    0.010482] IP: [<c102eeda>] cpu_attach_domain+0x6e/0x178
> [    0.010503] *pdpt = 0000000001454027 *pde = 0000000000000000
> [    0.010517] Oops: 0000 [#1] SMP
> [    0.010528] last sysfs file:
> [    0.010536] Modules linked in:
> [    0.010546]
> [    0.010554] Pid: 1, comm: swapper Tainted: G        W   
> 2.6.35-1-586-vyatta-virt #1 /
> [    0.010566] EIP: 0061:[<c102eeda>] EFLAGS: 00010206 CPU: 0
> [    0.010576] EIP is at cpu_attach_domain+0x6e/0x178
> [    0.010585] EAX: 0000042f EBX: c14eb868 ECX: 00000000 EDX: c14eb940
> [    0.010595] ESI: c14efb80 EDI: 00000000 EBP: c14eb9ac ESP: df035f40
> [    0.010605]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: e021
> [    0.010614] Process swapper (pid: 1, ti=df034000 task=df030000 
> task.ti=df034000)
> [    0.010625] Stack:
> [    0.010631]  c14eb940 0000102f 0000042f df044000 df006058 df006058 
> 00000000 c1447868
> [    0.010655] <0> 00000000 c102f634 00000027 c14eba08 00000000 00000020 
> c14479ac 00000001
> [    0.010683] <0> c1398564 00000000 c1398564 00000000 00000001 00000001 
> 00000001 00000000
> [    0.010713] Call Trace:
> [    0.010725]  [<c102f634>] ? __build_sched_domains+0x650/0x697

This might relate to BIGSMP.  Maybe something going wrong with the cpu
topology info?

What local changes do you have?  What kernel last worked for you?

Thanks,
    J

> [    0.010742]  [<c13df9e2>] ? sched_init_smp+0x81/0x11f
> [    0.010754]  [<c13cd326>] ? kernel_init+0xfd/0x1ba
> [    0.010764]  [<c13cd229>] ? kernel_init+0x0/0x1ba
> [    0.010776]  [<c1007d36>] ? kernel_thread_helper+0x6/0x10
> [    0.010784] Code: 42 38 89 44 24 04 8b 45 38 89 14 24 89 44 24 08 8d 45 5c 
> e8 94 a7 ff ff 8b 14 24 48 74 4c 8b 45 38 a9 8f 02 00 00 74 0b 8b 4d 08 <3b> 
> 09 0f 85 ef 00 00 00 a8 20 0f 85 e7 00 00 00 eb 2d 8b 4d 08
> [    0.010938] EIP: [<c102eeda>] cpu_attach_domain+0x6e/0x178 SS:ESP 
> e021:df035f40
> [    0.010955] CR2: 0000000000000000
> [    0.010965] ---[ end trace 4eaa2a86a8e2da25 ]---
> [    0.010988] Kernel panic - not syncing: Attempted to kill init!
> [    0.020000] Pid: 1, comm: swapper Tainted: G      D W   
> 2.6.35-1-586-vyatta-virt #1
> [    0.020000] Call Trace:
> [    0.020000]  [<c128954b>] ? panic+0x4a/0xbb
> [    0.020000]  [<c1038616>] ? do_exit+0x53/0x646
> [    0.020000]  [<c100a0b5>] ? oops_end+0x7d/0x80
> [    0.020000]  [<c1021cf8>] ? no_context+0x13b/0x144
> [    0.020000]  [<c102202d>] ? do_page_fault+0x0/0x24b
> [    0.020000]  [<c1021e06>] ? bad_area_nosemaphore+0xa/0xc
> [    0.020000]  [<c128b4a6>] ? error_code+0x66/0x6c
> [    0.020000]  [<c102202d>] ? do_page_fault+0x0/0x24b
> [    0.020000]  [<c102eeda>] ? cpu_attach_domain+0x6e/0x178
> [    0.020000]  [<c102f634>] ? __build_sched_domains+0x650/0x697
> [    0.020000]  [<c13df9e2>] ? sched_init_smp+0x81/0x11f
> [    0.020000]  [<c13cd326>] ? kernel_init+0xfd/0x1ba
> [    0.020000]  [<c13cd229>] ? kernel_init+0x0/0x1ba
> [    0.020000]  [<c1007d36>] ? kernel_thread_helper+0x6/0x10


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.