Re: [Xen-devel] XSA-180 follow-up: repurpose xenconsoled for logging

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 06/06/16 11:12, George Dunlap wrote:
> On 03/06/16 18:38, Andrew Cooper wrote:
>> On 01/06/16 15:00, Wei Liu wrote:
>>> Hi all
>>>
>>> During the discussion of XSA-180 Ian came up with the idea that we
>>> repurpose xenconsoled to handle logging. I've done some research (and
>>> some coding as well!).
>>>
>>> In my reply to George the other day:
>>>
>>>> I just read the code of virtlogd and xenconsoled.
>>>>
>>>> I think xenconsoled is missing at least things.
>>>>
>>>> From a higher level:
>>>>
>>>> 1. Abstraction of rotating file.
>>>> 2. Abstraction of client.
>>>> 3. IPC interface to libxl -- presumably we need to create a socket.
>>>>
>>> I've done #1 and port existing code to use that -- would be useful in
>>> general.
>>>
>>> #2 is not too hard because xenconsoled already has concept of a domain.
>>> I suspect some refactoring will be fine.
>>>
>>> #3 requires a bit more thinking. Virtlogd has an RPC interface -- I'm
>>> pretty sure we *don't* want that for xenconsoled. So I spent some time
>>> this morning and write up a draft for a xenstore based protocol. See
>>> below.
>>>
>>> Also there is an implication here: we put xenconsoled in a really
>>> critical path. If for some reason it doesn't work all guests are
>>> blocked. Do we really want to do this?
>> Sorry to be late to this thread, but I think all my Xen 4.7 tasks are
>> now done.
>>
>> Architecturally, this is a bad idea, and -1 from me.
>>
>> First of all, you are making the assumption that xenconsoled and all
>> qemus are in the same domain.  This is not necessarily the case,
>> although I suppose that `xl console` does depend on xl and xenconsoled
>> being in the same domain.
>>
>> At the moment, if xenconsoled crashes, the worst that happens is that
>> you cant interact with guest consoles, and the guests will notice that
>> their console rings aren't being drained.  This is a safe, albeit
>> suboptimal, situation which allows guests to continue running unimpeeded.
>>
>> If qemu is connected via xenconsoled, and xenconsoled crashes, qemu will
>> block.  Once qemu blocks, the VM is for all intents and purposes, dead. 
>> As you identify, there is a security implication here, where a guest
>> which can crash xenconsoled can also DoS all other HVM domains in the
>> system.
>>
>> From the scalability side, XenServer supports running 1000 windows VMs
>> on a single host (subject to sufficient RAM), and xenstored is a big
>> monlithic single point of failure.  We would like to avoid adding a second.
> qemu stubdoms is a red herring; that's a separate codepath which will be
> modified appropriately for its use case.

Most certainly not (and I didn't specify qemu stubdoms).  What happens
if someone wishes to run xenconsoled in a separate non-stub domain? 
What happens if someone has separate non-stub domains for running qemus?

All of these are currently viable options with XSM.

> The potential for qemu to
> block if the daemon dies, as well as scalability concerns, are good
> points which should be considered; and we certainly need to allow other
> ways of dealing with logging.
>
> But...
>
>> If someone builds Xen from source and finds later that their disks are
>> full up with logs, then tough.
> ...I cannot express how wrong-headed I think this statement is; and...

That is merely your opinion.  You know mine, and I stand by it.

>
>> If someone installs Xen from a distro,
>> they should reasonably expect logging, and log management, to be
>> consistent with other packages from the same distro.
> ...the degree to which this is true depends on the degree to which
> distributions are actually equipped to deal with these sorts of things.

A distro which isn't equipped to deal with these things is of no
practical use in the real world.

Any distro we care about supporting is equipped to deal with logs.

>
> I agree that if logging using existing logging systems were workable,
> that would be a better solution.  Wei did actually spend a decent chunk
> of time looking at other options, including logrotate and journald
> before settling on using xenconsoled.
>
> A lot of this brainstorming and discussion happened off-list because
> XSA-180 was still embargoed, so I can understand why it looks like this
> came out of nowhere.  It would probably be good for Wei to report here
> what he found and why he decided to propose this solution instead.

Please do.  Until there is an understanding of why the standard
mechanisms are not suitable, it is premature and naive to re-invent a wheel.

>
> FWIW, the libvirt project has exactly the same problem, and they did the
> analog of what Wei is proposing -- they added a new daemon, virtlogd, to
> handle all the console and debug log rotation in a fashion resistant to
> DoSing.  Without reading their discussion, it's reasonable to assume
> that using system logging was at least considered using system-level
> logging before deciding to write their own code.

Just because a project followed this path, doesn't mean it is a sensible
thing to do.

>
> We already have a daemon to do logging of consoles; it just doesn't have
> any of the logrotate features that are needed to make it robust against
> DoS.  There's no sense in having log rotation code in two places, so
> upgrading xenconsoled to do what virtlogd is doing makes more sense than
> say, either writing our own, or stealing virtlogd.

We have a daemon for connecting local pty's to guest consoles, which has
a disabled-by-default option to log consoles to files.

>
>> I think the best solution here is to manage expectations, not to follow
>> the kneejerk reaction and hack up a solution.
> Hopefully you can see now that this is not a knee-jerk solution. :-)
> Thank you for making sure other options is properly discussed in public.
>
> Tying off a few random technical threads:
>
> What does xapi do at the moment?

Plain syslog and logrotate.  No frills.

> Re the interface, it seems like having the option to log to {nothing,
> file, pipe, xenconsoled} should cover the reasonble usecases: libvirt
> for instance could take the pipe and pipe it to virtlogd (and xapi could
> do something similar if it wanted).

My main concern is that this is re-inventing a wheel which noone will use.

>
> Regarding qemu grinding to a halt if xenconsoled dies: That may be
> something worth considering; but at least on Linux you can set pipes as
> O_NONBLOCK and you'll get EAGAIN when writing to a full pipe instead of
> blocking.  We'd have to make sure that the error didn't cause any other
> failures (ignoring it and letting the write drop is probably the best
> thing), but that shouldn't be terribly difficult to do.

stderr really shouldn't be in a position where it gets -EAGAINs in a
common case.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel