Porting a new ARM platform to Xen ================================= ARM platforms are quite different one from each another. This can affect Xen porting: - interrupt controller (GIC on ARM) have different version; - interrupts numbers does not have many standards (beside CPU ones called SGIs); - every platforms have different addresses for devices; - devices list can be quite different. As a start you should read pages [1] and [2] from the wiki. One concept you have to learn is (at least currently) the device tree (DT for short, which can be found as source in .dts files in arch/arm/boot/dts Linux directory, .fdt or .dtb for compiled versions). Device tree is a specification that allow to describe device configurations. As device list is not standard DT contains description of all devices from memory to CPUs, system board, IO devices and so on. Information can be I/O ranges (in ARM I/O is memory mapped so there are not the port concept you can find on Intel), interrupts, device type, clock attached and many more. DT specification are quite extensible and allow for instance to have information like kernel command line too. You can find quite a lot of information on DT in Documentation/devicetree/bindings Linux directory (specifically there is an arm sub-directory). Another big difference is the boot process. Unfortunately is not still well defined as Intel one. For instance Xen use multiboot but multiboot is not still standard on ARM so can be quite painful. Happily Xen require very few devices in order to work, as stated in [2]: - GIC; - generic timers; - SMMU; - one UART. Now, fortunately timers are now quite standard and the interface to use them use CPU registers. UART ---- UART is the first device you should implement. The reason is simple, it allows you to see something from the beginning! Fortunately a lot of boards implement the standard (or compatible) 8250. At the beginning you should try to make the "early printk" work. Early printk is initialised very soon (just after few assembly lines). If you have a driver all you need is to setup memory ranges and device type and boot Xen. To make our UART work I just did this change to xen/arch/arm/Rules.mk (changeset 5ad71a6eefc9b09fe7458ded5b86b69fcc57436c): EARLY_PRINTK_BAUD := 115200 EARLY_UART_BASE_ADDRESS := 0x7ff80000 endif +ifeq ($(CONFIG_EARLY_PRINTK), hip04-d01) +EARLY_PRINTK_INC := 8250 +EARLY_PRINTK_BAUD := 115200 +EARLY_UART_BASE_ADDRESS := 0xE4007000 +EARLY_UART_REG_SHIFT := 2 +endif ifneq ($(EARLY_PRINTK_INC),) EARLY_PRINTK := y To enable early printk (after implementing the right code), you can have something like this in your .config: debug := y CONFIG_EARLY_PRINTK := hip04-d01 (hip04-d01 is the string decided for our platform). Early printk devices are implemented in xen/arch/arm/arm32 and xen/arch/arm/arm64 directories in files like debug-.inc. More documentation on docs/misc/arm/early-printk.txt file. Xen booting ----------- As said before starting Xen can be quite tricky. However to start you can focus on just the hypervisor. If you can boot Linux (and you really should make it work before trying Xen) you can replace the kernel with the Xen hypervisor (xen/xen file) and Xen will start. So you can test if serial and CPUs are working. Now that boot loader is configured and UART is working you should see Xen booting and try to do something! But without the FDT Xen won't do much. DTB file -------- Xen on ARM require a initial FDT that describe your hardware in order to handle your platform. If you were able to load the FDT while booting the Linux kernel you can use the same way and same FDT. Another way (which I used) is to embed the FDT file into Xen. In my .config file I have a line like CONFIG_DTB_FILE := $(XEN_ROOT)/hip04-d01.dtb this will embed hip04-d01.dtb file into Xen (in our case the DTB file was generated compiling Linux while on other cases is already provided by the firmware). Now with FDT sorted out you should see Xen doing some more progresses. Note: if you are using early printk after a while Xen will try to setup the serial again using the command line. In my case for some reasons at the beginning this setup caused serial to go silent, the temporary solution if to not pass a command line to Xen (which the default DT does not provide). Using this trick I was able to progress some more. This is the change to add the Xen command line to the DTS I made: chosen { linux,initrd-start = <0 0x10d00000>; linux,initrd-end = <0 0x12500000>; + xen,xen-bootargs = "console=dtuart dtuart=serial0"; }; See docs/misc/arm/booting.txt and docs/misc/arm/device-tree/booting.txt for more information. Platform -------- The next thing that probably will fail is the platform setting up. To initialise SMP you need some code specific to your platform un less your board support PSCI standard (you should find some psci string on dts files). Happily on xen/arch/arm/platforms there are many examples. Mainly the code require some tweak for setting up the additional CPUs. You can skip at the beginning the restart code but you should sort out processors initialisation. The other CPUs are started with a SGI (Software Generated Interrupt) but you also need to set the start entry of the CPUs. This is not standard and require specific code. To implement mine I looked at Linux sources and changesets for the platform. GIC --- Now if your GIC is compatible to standard ones (GICv2 or GICv3) you can just add your compatible string (it's a string you find in the DT to say the compatibility of each device) to the list and Xen should initialise and try to load the kernel. To add our GIC to the GICv2 list we did this change to xen/include/asm-arm/gic.h: #define DT_COMPAT_GIC_400 "arm,gic-400" #define DT_COMPAT_GIC_CORTEX_A15 "arm,cortex-a15-gic" #define DT_COMPAT_GIC_CORTEX_A7 "arm,cortex-a7-gic" +#define DT_COMPAT_GIC_HIP04 "hisilicon,hip04-intc" #define DT_MATCH_GIC_V2 DT_MATCH_COMPATIBLE(DT_COMPAT_GIC_CORTEX_A15), \ DT_MATCH_COMPATIBLE(DT_COMPAT_GIC_CORTEX_A7), \ - DT_MATCH_COMPATIBLE(DT_COMPAT_GIC_400) + DT_MATCH_COMPATIBLE(DT_COMPAT_GIC_400), \ + DT_MATCH_COMPATIBLE(DT_COMPAT_GIC_HIP04) #define DT_COMPAT_GIC_V3 "arm,gic-v3" Unfortunately our GIC was not fully compatible so it required changes to source (xen/arch/arm/gic-v2.c). Dom0 kernel ----------- If you reach this point you put the hypervisor in place of the kernel so where should Xen take the kernel to setup dom0? Remember: XEN IS NOT A BOOT LOADER. It has no driver for disk or network so the kernel must be already in memory, usually loaded by the boot loader. In my case I used another trick replacing the initrd with the kernel. As the boot loader was not working for me the firmware take kernel and initrd from specific part of board flash memory. So the trick was: - load kernel into initrd section of flash; - change FDT to point to initrd memory portion (same location used to provide initrd to kernel are used to provide kernel to xen). This is the change I made in DTS file to put the Linux kernel in the initrd partition: chosen { - linux,initrd-start = <0 0x10d00000>; - linux,initrd-end = <0 0x12500000>; + #address-cells = <1>; + #size-cells = <1>; + + xen,xen-bootargs = "console=dtuart dtuart=serial0"; + xen,dom0-bootargs = "console=hvc0 earlyprintk root=/dev/sda2 rw loglevel=7"; + module@0x10d00000 { + compatible = "xen,linux-kernel", "xen,multiboot-module"; + reg = <0x10d00000 0x300000>; + }; }; As you can see the start of module@0x10d00000 is the same of old initrd. Also note that the size of the area is smaller. This as Xen would try to load everything failing. Just put some value bigger than your kernel but not too much (0x300000 is 3 MB). Note that the kernel you boot inside Xen can be different from the kernel booted on bare metal. As [1] state ("Dom0 kernel" section): - you must enable Xen in the kernel configuration. See [3] for options to enable; - you must disable DTB inclusion if enabled. Xen do some tweak on the DTB in order to avoid conflicts. If kernel use just bare metal configuration bad things are going to happen. Debugging the kernel -------------------- Checking interrupts. Unfortunately I lost a lot of time (2/3 days) with interrupts. Linux was not getting interrupts due to a change in GICH registers. If you have a standard GIC probably you won't have this problem but is useful to understand the behaviour as interrupts can be very difficult to debug and sort out. Linux kernel was booting quite fine but suddenly it stopped. Using Xen console I was able to see register state and CPUs was idle. Before understanding that interrupts was not working I took a lot of debugging code and restarts. In ARM interrupts got mainly dispatched by a single function (code for my platform was in ./arch/arm/kernel/entry-armv.S). Basically a function registered with set_handle_irq is called (in my case by gic_init_bases in drivers/irqchip/irq-gic.c). Putting some debugging code here was the final prove for interrupts problems. Once the GIC was fixed dom0 start booting quite fine beside some small issues: - a small crash of some drivers as not included in the initial DTB, solved disabling it; - ethernet driver was not working, solved adding a call to dma_coerce_mask_and_coherent to set DMA bitmask correctly. I won't go into details on these part as they are more related to Linux kernel than to Xen. References ---------- [1] http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions [2] http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions_whitepaper [3] http://wiki.xen.org/wiki/Mainline_Linux_Kernel_Configs