[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] how to generate a smaller core with xm dump-core



å 2015/1/16 0:16, Don Slutz åé:
On 01/15/15 05:20, Ian Campbell wrote:
On Thu, 2015-01-15 at 11:31 +0800, Zhenzhong Duan wrote:
Hi Maintainers,


We are facing issue collecting  coredump using the xm dump mechanism
in Dom0.

We face couple of such issues daily, where the VMs panic s and the SA
team is supposed to collect the core.

The actual problem is that where the VMs are having huge RAM like 32+
GB RAMs dumping the core using xm dump generated core of almost the
same size.

And also when the RAM size is huge the time taken to dump is also
huge.

Transferring this core dump of this huge size to another system for
analysis is really a pain. I am looking for some solution where we can
generate smaller core using xm dump or any other tool.

Something similar to makedumpfile which eliminates the zero and
unnecessary pages, which results in core of smaller size.


This has become a hot issue and we terribly need this feature.

Can you come across with anything or are you aware of any feature
which could cater our needâ.?
I'm not aware of any existing features or efforts in this area.

If things like zero pages can be omitted from a core file without making
it non-standard then I think we'd be happy to see such patches to the
core libxc functions which produce the core files.

That wouldn't help much with a heavily loaded VM which is making good
use of all its ram though.

I've not looked but it's not unlikely that the dumping code itself is
not very optimal, so it might be worth having a look to see if more
batching etc might help speed things up at the root. Ages ago there were
patches floating around which improved the performance of the privcmd
mmap ioctl (from Mats Peterson), which might help here too (assuming
core dumping uses those interfaces as I expect). AFAIK those patches
were abandoned when Mats moved onto other things, resurrecting them
would probably help here as well as with migration and other
foreign-memory-man heavy workloads.

Other random thoughts:
       * Automatically compress the core file, doesn't help with time to
         produce the dump (quite the opposite) but helps with copying it
off box (but could also compress manually before moving or do it
         as a post processing step)
       * Have an option to produce some kind of "minicore", e.g. just
register state and a few pages of the referenced stacks, perhaps
         a few pages of RAM either side of any register which happens to
         be a valid looking pointer.
* Care would be needed to retain an acceptable probability
                 that the resulting core will actually be useful.
       * Consider including kernel RAM only (which might still be huge
         though if it has a 1:1 mapping of all RAM).

I don't think any of those would be suitable for enabling by default but
they might be useful to have in the arsenal.

HTH,

Ian.


If these are Linux guests then the patch:

http://lists.xenproject.org/archives/html/xen-devel/2013-11/msg02351.html

Can be used to enable crash to access the crashed guest and collect
some basic info. I would also include the output of xen-hvmctx and/or xenctx.
-- This is the quick way I would do Ian's minicore .

Note: dump-core currently does not include xen-hvmctx output (nor does it include
xenctx -a output).

Using the results from this may allow you to not need a copy of every dump ( not to imply that 2 similar minicore's would insure that a copy of each core
was not needed).

I also think that makedumpfile can process a xen dump-core file and make it
smaller.
Looks similar as gdbsx.
This patch will help if I could reproduce the issue locally.
But I can't access customer's env in most situation and they will not wait me to do online debug remotely.

Yes, makedumpfile may help in size rather than time here.
I am thinking if port some code from makedumpfile is possible and acceptable.

zduan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.