Re: [Xen-devel] Rumprun based upstream QEMU stubdom status update

[Wei Liu <wei.liu2@xxxxxxxxxx>]

Hi Eric

In fact we would like to have Linux-stubdom. Just that when it was
first posted back then we didn't know where to put it. Xen tree is not
an option. Now that Raisin is here it seems a good fit.

On Mon, Nov 23, 2015 at 02:52:47AM -0500, Eric Shelton wrote:
> Wei and others,
> 
> It has been a long, long road to getting a rumprun-based stubdom
> implemented.  The earliest discussion of pursuing this approach appears to
> be a post by IanJ on August 21, 2013, with the first serious indication of
> development work again posted by IanJ, on April 8, 2014 - over 19 months
> ago:
> 
> http://lists.xenproject.org/archives/html/xen-devel/2014-04/msg01066.html
> 
> From what is described below, it looks like there remains a long way to go,
> as what appears to be running is only printing out the output from "qemu
> --help" (pretty much what was described at XPDS15 more than three months
> ago).
> 
> Once upon a time, one of the release managers was talking about making QEMU
> upstream stubdom a blocker for 4.5:
> 
> http://lists.xen.org/archives/html/xen-devel/2014-04/msg01065.html
> 
> Obviously that did not happen.  Time and again, QEMU upstream stubdom has
> gotten shelved in favor of other priorities, which testifies as to where it
> falls on the Xen project's list of priorities.
> 
> I believe it is time to reassess the current development plan for QEMU
> upstream stubdom.  From a project management perspective, the current plan
> relies far too heavily on a very small handful of people that are simply
> too busy with other things to bring it to a working, let alone releasable,

Keep in mind that if it is the same set of people who are going to
work on Linux-based stubdom, the constraint is the same. Actually
there is more than just "making it work". I will elaborate on that
later.

> state.  Plus, the delay in realizing this has now reached a point where it
> imposes frequent, tangible, and possibly significant drains on development
> resources in order to cope with shortcomings of the current situation (QEMU
> upstream in dom0, and separate qemu-traditional codebase for stubdom).
> Plus, having QEMU upstream run in dom0 is likely Xen's single longest
> running identified security issue - and certainly the longest-lived one
> with an identified and actionable solution.  Consider how many of the
> QEMU-related XSAs reported something along the lines of "systems using
> qemu-dm stubdomain device models are NOT vulnerable."
> 
> Recently, following several XSAs demonstrating an interest in finding (and
> likely exploiting) QEMU vulnerabilities, the Xen team has devoted resources
> to running QEMU as non-root (now on patchset v10) - apparently trying to
> replicate the bandaid used by KVM.  This effort would not have been
> necessary if QEMU upstream stubdom had been implemented.  Additionally, the
> Xen team bears the burdens of maintaining the qemu-traditional tree,
> backporting fixes originating from various places into qemu-traditional,
> and having libxl cope with the differences between QEMU upstream and
> traditional and associated development headaches.  These represent
> underappreciated and unnecessary drain on the Xen project's limited
> resources imposed by not getting QEMU upstream stubdom implemented in a
> timely manner.
> 

While I agree current situation is not ideal, there is a flip side to
Linux-based or rump kernel based stubdom. Again, I will sum that up
later.

> Also, it looks like QEMU upstream would be the single most complex piece of
> software to be implemented under rumprun, which puts the current plan
> development way out on the bleeding edge.  It seems more like an experiment
> with lots of unknowns than a development project with a defined and
> predictable plan that simply needs man hours thrown at it.  Although
> rumprun has technical "sex appeal," and it taps into the community's recent
> interest in unikernels, rumprun still seems to be very immature at this
> time.
> 
> On top of that, getting this project implemented - even at a basic level of
> functionality - appears to require VERY specialized expertise held by VERY
> few people.  Is there really anyone other than Wei, IanJ, and Antti that
> has the skills needed to realize the current plan?  Further tightening the

Just to clarify some misunderstandings.

It's getting better at the moment. I think I managed to clear some
major obstacles (that is, writing drivers for rumprun, wrestling with
mini-os bugs or whatnot). It's in a better shape than before.

Another chunk of work is teaching QEMU to not initialised some
component or take a different path when initialising some
components. This is the same as Linux-based stubdom.

All that is left is plumbing toolstack and generating an image. Those
work items aren't really specific to rumprun based
stubdom. Linux-based stubdom is ahead in that regard, but there are
still patches waiting to be upstreamed and some other issues.

> resource crunch, those three are in very high demand for other projects:
> Wei had (and has?) release management duties, IanJ is involved in pretty
> much everything Xen, and Antti has broader pursuits to worry about.  The
> fact that no one on xen-devel has commented on Wei's October update in over
> a month indicates that he is essentially alone in his efforts.
> 
> 
> When does anyone _realistically_ see a rumprun-based QEMU reaching
> something approaching just a basic level of functionality as a stubdomain -
> let's say being able to run Windows 7 with a working display and networking?
>

There is no ETA at the moment TBH. I would only say as soon as
possible, when we find solutions to all external and internal issues
in this project. That includes but is not limited to doing the actual
work myself or waiting for external environment to become mature.

> 
> Unless the answer is along the lines of hitting at least tech preview
> status in Xen 4.7, I suggest that rumprun is not the right approach at this
> time.  I am fairly certain - based on my direct efforts in helping to move
> it along - that a Linux-based stubdom is something that CAN be realized in
> the 4.7 release timeframe.  Almost all of the problems you are trying to
> solve at this time with rumprun are already resolved today under Linux.
> Today, on a Linux-based QEMU upstream stubdom, you can SSH into a headless
> Linux distribution with fully working networking - this is light years
> beyond where rumprun is or will be any time soon.  Unsurprisingly, QEMU
> runs just fine in a Linux environment.  The only thing that remains to
> reach the "basic level of functionality" mentioned above is getting the
> display and keyboard/mouse input working, which I suspect will not be that
> big of an effort for someone familiar with QEMU upstream internals (and
> would have to be done anyway for rumprun).  I also believe that what has
> been described as the main problem for Linux-based stubdom - building a
> Linux-based image - is not as big as it has been made out to be: raisin may
> now reduce some of the previously perceived difficulties, and there is so
> much Linux expertise in the Xen community that I find it incredible that it
> is too hard to sort out how to build Linux on a Linux system that is
> already capable of building Xen and QEMU upstream.
> 

One lesson learned from QEMU-trad stubdom is that the build system
inevitably turns into a black hole that sucks in effort but yields no
clear return. That's why we would very much prefer doing things "the
right way", i.e. relying on well-established ways to deliver this
feature, and expanding the user base till it is large enough to draw
enough interest for this feature to be self-sustained.

What is really not clear at the moment is the path to Linux
distributions and BSDs. Xen project doesn't distribute binaries. Also
, to build an image requires specialised toolchain and build system
that distros are not interested in taking in. Building for *BSD is a
no-go at the moment for Linux-based stubdom and QEMU-trad stubdom.

That said, perfection is the enemy of good. I don't see a problem for
having Linux-based stubdom as a interim solution. As said, it was just
we didn't know where to put it.

> I am not suggesting that rumprun should be abandoned - just that in terms
> of actually getting QEMU upstream stubdom released and finally kicking
> qemu-traditional and other development resource drains to the curb, a
> Linux-based stubdom represents a practical and effective solution that can
> be used until rumprun is more mature and predictable.  Almost all of the
> work (with the exception of building the Linux-based image) is the same
> work that will have to be done to get the rumprun implementation working:
> getting the display working, and getting QEMU upstream to provide its
> current range of features from within a stubdom (as much of what is in
> there today depends on running in dom0).  So, when the day arrives that a
> rumprun-based stubdom is ready, little of what needs to be done for a
> Linux-based stubdom will end up being wasted effort.  Plus, there might
> just be some benefits from having QEMU running in a Linux PV domain
> (firewalling provided in the stubdom and easier debug, as possible
> examples) that could justify keeping it around even after rumprun is
> practical.
> 

I agree most of the things will be useful for both solutions.

>
> Thank you for taking the time to read this.  I have been following, and
> participating a bit in, this issue for more than 2.5 years, and I haven't
> observed any progress in that time - if anything, things are beginning to
> make backwards now that QEMU has become a target.  I believe it is time for
> the Xen project to acknowledge that a rumprun-based stubdom is beyond the
> level of resources and expertise that the project is actually willing or
> able to commit to it (as Xen has - quite rationally - set it aside time and
> again to pursue other matters).  In the face of finite and limited
> development resources, it is sensible to set aside a "perfect" solution
> aside, and instead pursue a path that has a genuine prospect of getting
> released sometime soon.
> 

Thanks for writing such long email. I see your point.

I discussed with Stefano and Anthony to reassess Linux-based stubdom,
with all the issues mentioned above, if we narrow down the scope of
the project -- ignore BSD support for now, integrate build with
Raisin, no distros integration -- it can actually be done within a
finite time frame.

Here are some issues with the existing patch series:

1. It's using Dracut build image, which is Redhat-centric.
2. Some patches are not yet complete and not suitable to upstream.

And these are the steps that need to be done:

1. Build with Raisin
  1.1 Build Linux kernel with a customised config.
  1.2 Build QEMU from a customised branch.
  1.3 Extract QEMU libraries dependency with generic tool.
  1.4 Generate a disk image with generic tool.
2. Toolstack plumbing and make work other components.
3. Teach OSStest to constantly test it.

Would you be up for working on Raisin integration?  That is, up to the
point of producing an image. We can discuss certain actions in
detailed and will help you with the issues you encounter along the
way. We appreciate help from the community as we're already quite
stretched with other projects.


Wei.

> All the best,
> Eric
>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel