############################################################## Installing Quick File Systems for New Domains in Xen 1.2 HOWTO Yan-Ching CHU 3 Feb 2004 ############################################################## Introduction ============ This is a small howto aiming at providing a step-by-step guide for Newbie of Xen to populate a file system for a new domain upon Xen so everything can be stored in and run from harddisk which is much faster than the Ramdisk + CDROM (/usr) approach intended for demonstration purpose. There are also some instructions of how to create new domain using the partition created by the new domain control tool available in Xen 1.2. Consult http://www.cl.cam.ac.uk/netos/xen for general information on Xen. This work is based on the following documents and aims as a supplement to them in the area of populating domains in Xen. Read them first! [1] Xeno Howto v1.1 by Bin Ren http://www.cl.cam.ac.uk/~br260/doc/xeno-1.1-howto.txt [2] README.CD (v1.0 & v1.2) by Dr. Ian Pratt http://www.cl.cam.ac.uk/Research/SRG/netos/xen/readmes/README.CD Preparation =========== We need the Xen 1.0 ISO image (xendemo-1.0.iso) from http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads/ and burn it into CD. It contains a working Redhat 9 distribution with that we can use it to populate a quick file system very easily without going through the installation procedure of those standard Linux distributions. The Xen 1.2 ISO will be released soon but it may not match the description here. Assume you have a Linux system like Redhat 9 with Xen 1.2 running. If not consult [1,2], and get the sources, build Xen and Xenolinux 1.2 as well as the control tools by following almost the same procedure as in [1]. You can download the tarball of Xen 1.2 (xeno-1.2.tar.gz) from the above link again, or use the bk download method as stated in [1]. Create New Partitions for Domains ================================= You can't use the same root of Domain 0 for domains>0. Multiple writers to a single file system will result in one that can't be repaired. There are serveral approaches that one can use for multiple domains (in harddisk) on Xen. The simplest one is just to use a complete root file systems (including / and /usr) in each domain. Another one is to use a shared Read-only /usr for multiple domains, this can save a lot of space but changing the /usr later will affect all related domains. One can also use NFS approach stated in [2]. Here only the simplest one is described. More may be written later. For the whole / and /usr from the ISO, you will need about 1350MB for each partition and you may want to create partition with space more than that for other purposes. Note fdisk create new harddisk partition with a new largest partition number so it won't affect the current settings, particularly the boot process. If you don't have enough space you may want to use tools like Partition Magic to resize a current partition and create new partition. This may cause all the partition numbers being shifted especially if you use Partition Magic in windows. GRUB may not be able to locate the original config file and you may not be able to boot up normally. Don't panic, follow the instructions below (bottom). Create a new partition by fdisk (eg. using the 1st IDE harddisk): # fdisk /dev/hda Then type 'n' and enter the size you want ('m' for menu & 'p' for current partition table), when finished and confirmed enter 'w'. You need a reboot for the new setting to take effect. Make a file system (ext3) on it (eg. hda12) after the partition table is updated after reboot: # mkfs -t ext3 /dev/hda12 Populate the Partition with Redhat 9 from the ISO ================================================= Create a mount point for the new partition and mount it: # mkdir /mnt/domain1 # mount /dev/hda12 /mnt/domain1 Mount the CDROM drive with appropriate device name: # mount -t iso9660 /dev/hdb /mnt/cdrom To install the ROOT system, untar the file (root.tar.gz) from the ISO: # cd /mnt/domain1 && tar zxvpf /mnt/cdrom/XenDemoCD/root.tar.gz Note the directory "bin", "lib" and "sbin" are all symbolic links to /root in the cdrom because it was to provide a read-only /usr in the cdrom for sharing. That complicates matters and in a harddisk installation we can get back to normal: # rm -f bin boot lib sbin # cd /mnt/cdrom/root && cp -a bin lib sbin /mnt/domain1 To install the /usr system, copy everything in the disc except the directory "XenDemoCD", "boot" and "root". Note they altogether are very big (~1.3GB) in size and it may take a while. Also note there are some directories you may not want in "local" such as the "local/src" and other irrelevant files, this may save a bit space. If so, leave out "local" and copy the things you want later. # cd /mnt/cdrom && cp -a X11R6 etc java libexec src bin dict kerberos local sbin tmp doc include lib man share /mnt/domain1 We are going to boot the new domain by using the same Xenolinux kernels as in Domain 0 so we don't need a /boot in it. In case you want to use other kernels remember to specify the location of it, see the section below. Modify "/mnt/domain1/etc/fstab" in the domain so it reflects your system's settings. Also you may want to modify "passwd" file, if not the root account and password would still be "root" and "xendemo" respectively. Edit a Customized Config File for a Domain ========================================== Because there is a new set of domain control tools in Xen 1.2 the way you create new domains is different from [1], but is detailed in [2]. Assume you have installed the Xen tools, if not consult [1,2] again. In short: # cd /usr/src/xeno-1.2.bk/tools && make && make install The binaries are installed in /usr/bin. To create new domains we need to use "xc_dom_create.py". This reads the default file in /etc/xc called "defaults". "xc_dom_create.py" uses /etc/xc as default directory and you can have other config files there. When you specify a file name as argument it will override the defualt file. Only the argument "vmid" is necessary to be passed through command line and the other variables can be specified in the config file beforehand. Copy "defaults" to another config file, for that you may want to refer to it later: # cd /etc/xc && cp defaults domain1.conf Edit "domain1.conf" - Step 1 - Change the image file location, suppose using the same as Domain 0: image = "/boot/xenolinux-1.2.gz" Step 2 - VM memory Size, normally 32, 48, 64, 96, 128 are all handy. mem_size = 64 Step 3 - Just the VM name, you can ignore it. Step 4 - You can leave it or if your ip address is public and don't want the new domain to use it. You may want to change to only local link address: vfr_ipaddr = [ XenoUtil.add_offset_to_ip('169.254.1.0',vmid) ] Step 5a - Granting physical access to devices to VM - very important! eg, hda12 is your new domain partition vbd_list = [ ('phy:hda12','hda12','w') ] Step 5b - Leave it zero. Step 6 - Set netmask, gateway, NFS server address, runlevel etc. netmask = '255.255.0.0' gateway = '169.254.1.0' nfsserv = '169.254.1.0' Note using runlevel 4 can avoid many ugly error messages generated when the VM kernel tries to access resources that is available to Domain 0 only, such as accessing console with "mingetty". Setting root as read-only first can avoid fsck to complaint; the root will be mounted again as read-write after the initial boot process. By the way, turn eth0 on. cmdline_ip = "ip="+vfr_ipaddr[0]+":"+nfsserv+":"+gateway+":"+netmask+"::eth0:on" cmdline_root = "root=/dev/hda12 ro" cmdline_extra = "4 VMID=%d" % vmid Step 7 - Leave it. Start the new Domain with a customized config file ================================================== Enable NAT in Domain 0 for communication with other domains>0. # xen_nat_enable Open another terminal and key in this to allow domain0 to read the console output by other domains through UDP: # xen_read_console & Remember to umount the new partition in use. Leave the directory /mnt/domain1 if you are under it or you will not be able to umount it. # umount /dev/hda12 Now everything should be ready, create the new domain with "xc_dom_create.py" with your own config file overriding "defaults": # xc_dom_create.py -fdomain1.conf -Dvmid=1 You can override other variables in the config file without modifying it, for example, you want to have larger memory size (see [2] for more details): # xc_dom_create.py -fdomain1.conf -Dvmid=1 -Dmem=128 You should be able to see the new domain booting in the terminal that you specified "# xen_read_console &". SSH into the new domain by its IP, which is dynamic according to the actual domain ID - it may not necessarily be the "vmid" you entered. ssh root@xxxxxxxxxxx Remember to try the new domain control tools "xc_dom_control.py". Simply type it without arguments will get you the help menu. Read all domains' info by: # xc_dom_control.py list Add More VM Domains =================== Simply follow the instructions above again with little adjustments you should be able to create many more domains and get them running. Tips: By using multiple config files for domains will make life a lot easier. Use aliases for the "xc" commands helps a lot because their names are a bit long. Typing the whole domain creation command, for example, # xc_dom_create.py -fdomain1.conf -Dvmid=1 looks tedious if you want to try something many times. Write your own shell scripts to automate this. Manually Control GRUB ===================== If you messed up the device partition numbers when you create new partitions you may not be able to see the GRUB boot manual normally. Don't panic, here is a simple way to work around this. Try make an educated guess of where the kernel is located. If you were using hda6 for the kernel originally, depending how many partitions you have added which have affected the partition number, calculate the likely new partition number. For example, if just one partition has been added, the new partition number for hda5 should be hda4/hda6. Note GRUB uses index starting from 0. "hda5" is denoted as (hd0,4) - first harddisk partition 5 if start from 0. Suppose you are in the interactive menu in GRUB (even you messed the partition number up GRUB is still in the Master Boot Record). Specify your root by: > root (hd0,5) or if it doesn't work later try another: > root (hd0,3) Now comes the turn to specify kernel location. Normally the name is in the form vmlinuz-'version_number', try: > kernel /boot/vmlinuz-2.4.22 If it can't find the file it's rather the root is wrong or the file name is wrong, try until you get it right. Once the kernel can be found enter: > boot Suppose you get through this and get Linux running again. Modify /boot/grub/grub.conf to reflect the current setting with appropriate partition numbers. Run the command below to have GRUB recognizing the current root again: # /sbin/grub-install /dev/hda You should be ok after reboot.