Re: [Xen-API] help on - massive logging in /var/log/messages hang XCP VMs

[John Else <john.else@xxxxxxxxxx>]

(cc’ing xen-api)

 

Hi Kiran,

 

It would technically be possible to modify the fe daemon to allow the option of redirecting qemu’s output to an arbitrary location, but plumbing this through to the xen-api level to make it usable would be a lot of work, and since this is a very unusual use case I don’t think it’s likely to happen.

As I said in #xen-api, I think you’d be better off writing this data to a disk mounted on the guest – transferring this much information really isn’t what the guest -> qemu -> syslog data path is designed for.

 

John

 

 

From: kiran pawar [mailto:kinpaa@xxxxxxxxx]
Sent: 20 December 2012 11:28
To: John Else
Subject: Re: help on - massive logging in /var/log/messages hang XCP VMs

 

Hey John,

 

I installed fix http://support.citrix.com/article/CTX135235

 

It allow me to redirect debug log to some extent. It still hang if I redirect with huge data e.g. 10 MB/second. I believe this fix contain same patch you pointed out.

Is there anything we can do in fe daemon to make it work for more log redirection ?

 

~Kiran

 

On Wed, Dec 19, 2012 at 9:12 PM, kiran pawar <kinpaa@xxxxxxxxx> wrote:

Hi John,

logs are coming from qemu only. I believe issue I am seeing can be reproed if someone redirect heavy logs from qemu to syslog.

Is there any solution to this ?  I am using workaround which is annoying as I have to open terminal for each VM.

 

=============================

ls /proc/<qemu_vm_log_id>/fd/2 will show the reader of the pipe. 

l-wx------ 1 root root 64 Oct 7 18:49 2 -> pipe:[340027] 

[root@ckinpaa]# lsof | grep 340027 
fe 14010 root 4r FIFO 0,6 0t0 340027 pipe 
qemu-dm.b 14011 root 1w FIFO 0,6 0t0 340027 pipe 
qemu-dm.b 14011 root 2w FIFO 0,6 0t0 340027 pipe 

so, if I just do a "cat /proc/<vmiop_log_id>/fd/2", the program will continue. 

============================

 

~Kiran

 

On Wed, Dec 19, 2012 at 9:02 PM, John Else <john.else@xxxxxxxxxx> wrote:

Hi Kiran,

 

The difference is to with how XCP launches qemu – it is launched via fe (the fork-exec daemon) which then redirects all output from qemu via syslog to /var/log/messages.

 

Does your large amount of logging come from the guest via qemu, or another source? If it’s via qemu, you could be running into the bug I mentioned. If it is via another source, but still using syslog, then it could be that syslog is so busy it’s not able to read logging from qemu, and qemu hangs. Unfortunately I don’t think it would be possible to redirect qemu’s output to anything other than syslog without modifying the fe daemon itself.

 

John

 

From: kiran pawar [mailto:kinpaa@xxxxxxxxx]
Sent: 19 December 2012 15:19
To: John Else
Subject: Re: help on - massive logging in /var/log/messages hang XCP VMs

 

John,

I am working on project where plugin is created to share some information between host and guest. This plugin echos lot of messages to log.

In open Xen, messages from plugin comes to /var/log/xen/qemu-dm-xxx.log while in XCP those are coming to /var/log/messages only. 

 

Is there any reason output from qemu can be blocked ? Can I redirect VM specific log to different log files in XCP ? I am newbie to this things.

 

Thanks,

~Kiran

On Wed, Dec 19, 2012 at 7:29 PM, John Else <john.else@xxxxxxxxxx> wrote:

Hi Kiran,

 

Hmm, I guess you might be seeing a slightly different but related problem – the output from qemu may be blocked by syslog being busy, rather than by the problem with FE which is fixed by that patch. This would cause the VM to hang in a similar way.

Why is so much logging going to /var/log/messages? Are you not able to redirect it somewhere else?

 

John

 

From: kiran pawar [mailto:kinpaa@xxxxxxxxx]
Sent: 19 December 2012 13:12
To: John Else
Subject: help on - massive logging in /var/log/messages hang XCP VMs

 

Hi John,

I am using XCP 1.5. When I redirect massive logs 10 MB/second to /var/log/messages, VM hang.  If I do cat /proc/<qemu-vm-process-d>/fd/2, VM continues. 

 

As suggested on freenode, I applies patch https://github.com/xen-org/xen-api-libs/commit/a13e0d968003e087bf40f4362f866915ba68aeda and then installed xapi-libs-fe-xxx.rpm. But this does not solve issue.

 

Can you please provide some help on this issue ?

 

Thanks,

~Kiran

 

 

 

_______________________________________________
Xen-api mailing list
Xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api