[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Troubles analyzing crash dumps from xl dump-core



On 30.01.21 19:53, Roman Shaposhnik wrote:
On Fri, Jan 29, 2021 at 11:28 PM Jürgen Groß <jgross@xxxxxxxx> wrote:

On 29.01.21 21:12, Roman Shaposhnik wrote:
Hi!

I'm trying to see how much mileage I can get out of
crash(1) 7.2.8 (based on gdb 7.6) when it comes to
analyzing crash dumps taken via xl dump-core (this
is all on x86_64 with stock Xen v. 4.14).

The good news is that the image actually does load
up but it throws the following WARNINGs in the process:

WARNING: cannot access vmalloc'd module memory
crash: read error: kernel virtual address: ffffffff93613480  type:
"fill_task_struct"
WARNING: active task ffffffff93613480 on cpu 0 not found in PID hash
crash: read error: kernel virtual address: ffffffff93613480  type:
"fill_task_struct"
WARNING: cannot read log_buf contents

And then the info that it gives me around basic things like
ps, mod, log, etc. is really super limited (and I am now suspecting
may even be wrong).

Since I was primarily after dmesg/log initially, I tried:
crash> log
log: WARNING: cannot read log_buf contents

Then I tried taking an xl dump-core from the domU that was still
very much alive and happy and got similar results -- so it clearly
doesn't seem to be related to the state domU is in.

As matter of fact, I actually got to the desired dmesg output
by simply running strings(1) on the core file -- so the info is
definitely there -- but I guess some kind of index/reference maybe
broken.

With all that in mind, if there's anyone on this ML who has recently
done Xen DomU crash dump analysis -- I would definitely appreciate
the pointers!

For me it just works (openSUSE).

Can you please run:

crash --version and readelf -a XXXX (on the xl dump-core output)
and post the results?

# crash --version

crash 7.2.1
Copyright (C) 2002-2017  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".

# readelf -a pv-dump
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 01 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       1
  Type:                              CORE (Core file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          64 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         0
  Size of section headers:           64 (bytes)
  Number of section headers:         7
  Section header string table index: 1

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .shstrtab         STRTAB           0000000000000000  40404000
       0000000000000048  0000000000000000           0     0     0
  [ 2] .note.Xen         NOTE             0000000000000000  00000200
       0000000000000568  0000000000000000           0     0     0
  [ 3] .xen_prstatus     PROGBITS         0000000000000000  00000768
       0000000000002860  0000000000001430           0     0     8
  [ 4] .xen_shared_info  PROGBITS         0000000000000000  00002fc8
       0000000000001000  0000000000001000           0     0     8
  [ 5] .xen_pages        PROGBITS         0000000000000000  00004000
       0000000040000000  0000000000001000           0     0     4096
  [ 6] .xen_p2m          PROGBITS         0000000000000000  40004000
       0000000000400000  0000000000000010           0     0     8
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  l (large), p (processor specific)

There are no section groups in this file.

There are no program headers in this file.

There is no dynamic section in this file.

There are no relocations in this file.

The decoding of unwind sections for machine type Advanced Micro Devices X86-64 is not currently supported.

No version information found in this file.
No note segments present in the core file.


also, what version of  Xen are you using?

Current staging branch.


I tried a pv guest only with a 4.4 kernel, though.

I would really appreciate it if you could try it on HVM so that
I know whether it is an issue that only affects my setup or
other folks (especially after seeing answers to above).

Works as well.

# readelf -a hvm-dump
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 01 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       1
  Type:                              CORE (Core file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          64 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         0
  Size of section headers:           64 (bytes)
  Number of section headers:         7
  Section header string table index: 1

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .shstrtab         STRTAB           0000000000000000  8044f260
       0000000000000048  0000000000000000           0     0     0
  [ 2] .note.Xen         NOTE             0000000000000000  00000200
       0000000000000568  0000000000000000           0     0     0
  [ 3] .xen_prstatus     PROGBITS         0000000000000000  00000768
       0000000000001430  0000000000001430           0     0     8
  [ 4] .xen_shared_info  PROGBITS         0000000000000000  00001b98
       0000000000001000  0000000000001000           0     0     8
  [ 5] .xen_pages        PROGBITS         0000000000000000  00003000
       000000008004c000  0000000000001000           0     0     4096
  [ 6] .xen_pfn          PROGBITS         0000000000000000  8004f000
       0000000000400260  0000000000000008           0     0     8
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  l (large), p (processor specific)

There are no section groups in this file.

There are no program headers in this file.

There is no dynamic section in this file.

There are no relocations in this file.

The decoding of unwind sections for machine type Advanced Micro Devices X86-64 is not currently supported.

No version information found in this file.
No note segments present in the core file.


Juergen

Attachment: OpenPGP_0xB0DE9DD628BF132F.asc
Description: application/pgp-keys

Attachment: OpenPGP_signature
Description: OpenPGP digital signature


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.