Re: [Xen-users] linux stubdom

[Ian Campbell <Ian.Campbell@xxxxxxxxxx>]

On Tue, 2013-01-29 at 19:32 +0000, Markus Hochholdinger wrote:
> Hello,
> 
> Am 29.01.2013 um 17:36 Uhr schrieb Ian Campbell <Ian.Campbell@xxxxxxxxxx>:
> > On Tue, 2013-01-29 at 15:46 +0000, Markus Hochholdinger wrote:
> > > I'm reading a lot about xen stub-domains, but I'm wondering if I can use
> > > a linux stubdom to serve a "transformed" block device for the according
> > > domU. The wiki page(s) and the stubdom-directory in the source code
> > > leaves a lot of questions to me I'm hoping someone can answer me here.
> > I think the thing you are looking for is a "driver domain" rather than a
> > stubdomain, http://wiki.xen.org/wiki/Driver_Domain. You'll likely find
> > more useful info if you google for that rather than stubdomain.
> 
> a driver domain is a great thing, but I'm wondering if I can live migrate a 
> domU inclusive its driver domU? In the domU config I have to use the $domid 
> of 
> the driver domain. This $domid is probably wrong after a live migration of 
> the 
> domU and I have to migrate the driver domU in the same time while migrating 
> the domU. If the driver domU is ment to stay on one dom0 there's no 
> difference 
> between doing this in dom0, but for me this doesn't work.

The change of $domid doesn't matter since a migration involves
reconnecting the devices anyway, which means thy will reconnect to the
"new" driver domain.

The normal way would be to have a driver domain per host but there's no
reason you couldn't make it such that the driver domain was migrated too
(you'd have to do some hacking to make this work).

AIUI you currently have a RAID1 device in the guest, presumably
constructed from 2 xvd* devices? What are those two xvda devices are
backed by? I presume it must be some sort of network storage (NFS,
iSCSI, NBD, DRDB) or else you just couldn't migrate.

Are you intending to instead run the RAID1 device in a "driver domain",
constructed from 2 xvd* devices exported from dom0 and exporting that as
a single xvd* device to the guest? Or are you intending to surface the
network storage directly into the driver domain, construct the RAID
device from those and export that as an xvd* to the guest?

> Do you know how I can live migrate a domU which depends on a driver domU? How 
> can I migrate the driver domU?
> For my understanding the block device has to be there on the destination dom0 
> before live migration begins but is also used on the source dom0 from the 
> migrating, but still running, domU.

Not quite, when you migrate there is a pause period while the final copy
over occurs and at this point you can safely remove the device from the
source host and make it available on the target host. The toolstack will
ensure that the block device is only ever active on one end of the other
and never on both -- otherwise you would get potential corruption.

While you could migrate the driver domain during the main domU's pause
period it is much more normal to simply have a driver domain on each
host and dynamically configure the storage as you migrate.

> Can I combine a driver domU to a normal domU like I can combine a stubdom 
> with 
> a normal domU?

If you want, it would be more typical to have a single driver domain
providing block services to all domains (or one per underlying physical
block device).

> I thought a stubdom live migrates with its domU so you don't have to worry 
> that the driver domU live migrates while the according normal domU migrates.

A stubdom is a bit of an overloaded term. If you mean an ioemu stub
domain (i.e. the qemu associated with an HVM guest) then a new one of
those is started on the target host each time you migrate.

If you mean a xenstored stubdom then those are per host and are not
migrated.

And if you mean a driver domain then as I say those are usually per host
and the domain will be connected to the appropriate local driver domain
on the target host. 

> > > So my question are:
> > > * What are the requirements to run linux inside a stubdom? Is a current
> > > pvops
> > >   kernel enough or has the linux kernel to be modified for a stubdom? If
> > >   yes, I would prepare a kernel and a minimal rootfs within an initrd to
> > >   setup my blockdevice for the domU.
> > You can use Linux as a block driver storage domain, yes.
> 
> OK, I see. But still, would it be possible to run linux in a stub-domain? 
> I've 
> read e.g. http://blog.xen.org/index.php/2012/12/12/linux-stub-domain/ which 
> describes this, but I'm unsure if this will be (or is already) supported by 
> current xen?

This work is about a Linux ioemu stub domain. That is a stubdomain with
the sole purpose of running the qemu emulation process for an HVM
domain. I think the intention is for this to land in Xen 4.3 but it does
not have anything to do with your usecase AFAICT.

Everything you want to do is already possible with what is in Xen and
Linux today, in that the mechanisms all exist. However what you are
doing is not something which others have done and so there will
necessarily need to be a certain amount of putting the pieces together
on your part.

> > > * How can I offer a block device (created within the stubdom) from the
> > > stubdom
> > > 
> > >   to the domU? Are there any docs how to configure this?
> > > 
> > > * If the above is not possible, how could I offer a blockdevice from one
> > > domU
> > > 
> > >   to another domU as block device? Are there any docs how to do this?
> > Since a driver domain is also a domU (just one which happens to provide
> > services to other domains) these are basically the same question. People
> > more often do this with network driver domains but block ought to be
> > possible too (although there may be a certain element of having to take
> > the pieces and build something yourself).
> 
> Is live migration possible with this driver domUs? What requirements are 
> needed that I can live migrate a domU which depends on a driver domU?
> 
> 
> > Essentially you just need to a) make the block device bits available in
> > the driver domain b) run blkback (or some other block backend) in the
> > driver domain and c) tell the toolstack that a particular device is
> > provided by the driver domain when you build the guest.
> 
> Yes, I would provide two block devices (logical volumes) into the driver 
> domU, 

How are you doing this? Where do those logical device come from and how
are they getting into the driver domU?

> create there a software raid1 device and make the md device available with 
> blkback. I would do this for each domU so I can live migrate domUs 
> independent. The driver domU only needs a kernel and a initrd with a rootfs 
> filled with enough to build the md device and export it with blkback.
> 
> But how can I address this exported block device? As far as I've seen I need 
> the $domid of the driver domain in my config file of the domU, or am I 
> missing 
> something?

$domid can also be a domain name, and you can also change this over
migration by providing an updated configuration file (at least with xl).

> So this would be a dependency that the driver domain is started before the 
> stubdom with qemu.

Yes.

> In some subdom-startup script I saw the parameter "target" while creating the 
> stubdom. Is this a possibility to combine two domUs?

I think "target" in this context refers to the HVM guest for which the
ioemu-stubdom is providing services.

> As far as I understand with a driver domU I need the $domid of the driver 
> domU 
> in my config, so this is the connection between the two domUs.
> But with stub-domains I haven't understand how data flows between stubdom and 
> domU and because I've seen a lot of nice little pictures describing that I/O 
> flows between domU over stubdom to dom0 and backwards I thought stub-domains 
> would be the way to go.

Only if you are using emulated I/O. I assumed you were using PV I/O, is
that not the case?

Ian.


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users