Re: [Xen-users] Migrating domUs behind a firewall backend domU

[Jan Behrend <jbehrend@xxxxxxxxxxxxxxxxx>]

On Monday 14 April 2008 13:54, Jan Behrend wrote:
> I have the following setup:
>
> - standard Dom0, _without_ xen bridge
> - netbackend domU as a firewall, i.e. nic is exclusively used by this domU
> via pciback mechanism. This domU hosts the xen bridge and does the network
> setup for the other domUs (/etc/xen/scripts/vif-bridge etc.)
>
> - a bunch of domUs _behind_ this firewall domU
>
> This setup is described in
> http://lists.xensource.com/archives/html/xen-users/2005-07/msg00558.html
>
> Everything is working just fine. So far so good.
> My next goal is to migrate one of the domUs _behind_ the netbackend
> firewall to another machine with the same setup.  I know I cannot migrate
> the netbackend firewall domU.
>
> The migration works fine but unfortunately networking stops completely on
> the migrated domain. Otherwise it works just fine but has no network. This
> is what happens:
>
> On xenhost1 I do:
> # xm list
> # xen16                                     15      128     1 -b----    
> 14.4
>
> # xm migrate xen16 xenhost2 --live
> #
>
> This is what's then shown on xenhost2:
> # xm list
> # xen16                                     14      128     1 -b----     
> 0.0
>
> The following is the kernel output on the console.  When I hit enter I see
> the login screen again:
> # xm console xen16
> vif vif-0: 2 reading other end details from device/vif/0
> xenbus: resume (talk_to_otherend) vif-0 failed: -2
>
> Both Xenhosts are 64bit Debian Etch (Xen 3.0.3).  The common disk is hosted
> on an iSCSI target. Kernelversion is 2.6.18-5-xen-amd64 on all of the dom0s
> and domUs.
>
> This is the xen16.cfg on both Xenhosts:
> ***snip***
> kernel  = '/boot/vmlinuz-2.6.18-5-xen-amd64'
> ramdisk = '/boot/initrd.img-2.6.18-5-xen-amd64'
> memory  = '128'
> maxmem  = '256'
> root    = '/dev/sda1 ro'
> disk    =
> [ 'phy:/dev/disk/by-uuid/10875585-5295-4b1c-9043-3c87d052a3e7,sda1,w',
> 'phy:/dev/disk/by-uuid/b993558c-933c-4391-a7b8-c6904f934b0d,sda2,w' ]
>
> name    = 'xen16'
> vif  = [ 'mac=00:16:3E:40:53:23 , backend=netbackend' ]
> on_poweroff = 'destroy'
> on_reboot   = 'restart'
> on_crash    = 'restart'
> ***snip***
>
> I have the feeling that the migrated domU just does not get connected to
> the netbackend domU.  Because there are no network packets caught on either
> fo the firewall backends, I believe that firewall rules are not the
> problem. (I even flushed them to no avail)
>
> Anybody having thoughts on this?  Need more information?
>
> Cheers Jan Behrend

Hot Diggity, I made it!

Finally migration in the scenario described above works.
First of all I upgraded to Xen 3.2.  The downtime during live migration with 
this method is not milliseconds but rather a couple of seconds.  This is 
because the network device needs to be reattached to the netbackend.
Other than this it works beautifully.   Here it goes:

This describes a live migration of the xen16 DomU from Dom0 Xenhost1 to Dom0 
Xenhost2

1. Detach the network device from the DomU on Xenhost1
2. Take down the network in the DomU
3. do the actual migration from Xenhost1 to Xenhost2
4. Attach the network device to the DomU on Xenhost2
5. Bring up the network device in the DomU

Done!
If you use a script, pings to the DomU are disrupted for 1 to 5 seconds 
depending on the DomU's RAM size.

Since this is all done via network, taking down the network interface in the 
domU leaves you with the problem of bringing it up again via network.  The 
trick is to look for the "/sys/class/net/eth0" which disappears while 
detaching the network device in the Dom0 and reappears while attaching the 
network device in the other Dom0 once the migration has taken place.

Here is the script I wrote to do this fairly quickly.  This is not failsafe, 
so you might wat to add safety measures to the it.  Anyway, this is just 
meant as an inspiration. You'll need passwordless ssh logins on all machines.  
I use Kerberos for this.

#!/bin/bash
### Fix the line breaks which occured while posting to the list ###

function usage {
   echo "usage: ${0##*/} <Domain> <Host>"
   exit 0
}

if [ $# -ne 2 ] ; then
    usage
fi

if [ ! $(xm domid $1) ] ; then
   echo "Dying ..."
   exit 1
fi

echo "Timing network wakeup for DomU \"$1\""
ssh $1 'while [ -d /sys/class/net/eth0 ] ; do sleep 1 ; done ; ifdown eth0 ;  
until [ -d /sys/class/net/eth0 ] ; do sleep 1 ; done ; ifup eth0' &
echo "Detaching network device from DomU \"$1\""
MAC_ADDR=$(xm network-list $1 | tail -n1 | awk '{print $3}')
for fn in $(xm network-list $1 | tail -n-1 | awk '{print $1}') ; do
   xm network-detach $1 $fn -f
done
echo "Actually migrating \"$1\" to \"$2\""
xm migrate $1 $2 -l
echo "Attaching network device to DomU \"$1\""
ssh $2 "xm network-attach $1 mac=$MAC_ADDR backend=netbackend"
while [ $(ssh $2 "xm network-list $1" | tail -n1 | awk '{print $5}') -ne 4 ] ; 
do
   echo "Attaching network device to DomU \"$1\" FAILED!!! Retrying ... (Kill  
me with C-c if you wish!)"
   for fn in $(ssh $2 "xm network-list $1" | tail -n-1 | awk '{print $1}') ; 
do
      ssh $2 "xm network-detach $1 $fn -f"
   done
   ssh $2 "xm network-attach $1 mac=$MAC_ADDR backend=netbackend"
done
echo "Done! (Disregard SIOCDELRT error message)"
echo

Cheers Jan
-- 
Jan Behrend
Max-Planck-Institut fuer Radioastronomie
Abteilung fuer Infrarot-Technologie
Auf dem Huegel 69, D-53121 Bonn (Germany)
Tel: (+49) 228 525 319, Fax: (+49) 228 525 411
jbehrend@xxxxxxxxxxxxxxxxx http://www.mpifr-bonn.mpg.de

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users