Re: [Xen-devel] Problems running Xen 4.4 on OMAP5432 evm

[Chen Baozi <baozich@xxxxxxxxx>]

On Apr 28, 2014, at 20:16, Chen Baozi <baozich@xxxxxxxxx> wrote:

> 
> On Apr 27, 2014, at 14:30, Kapania, Ashish <akapania@xxxxxx> wrote:
> 
>> Hi Baozi,
>> 
>> I managed to resolve the issue of dom0 aborting. The linux kernel I am
>> using (v3.8.13 that comes with TI's GLSDK) had "zreladdr" set to
>> 0x80008000 which was causing the kernel to load the page table at
>> 0x80004000 (address not mapped by xen stage2 mmu table). To fix
>> the issue, I enabled "CONFIG_AUTO_ZRELADDR" in the config and
>> loaded the kernel image to 0xC0800000.This moved the page table
>> from 0x80004000 to 0xCxxxxxxx address range which is mapped by xen.
> 
> I test with the following u-boot cmd:
> 
> # fatload mmc 0:1 0x825f0000 <dtb-name>
> # fatload mmc 0:1 0x90000000 <xen uImage name>
> # fatload mmc 0:1 0xa0000000 <dom0 zImage name>
> # bootm 0x90000000 - 0x825f0000
> 
> Note that the address zImage loaded to should be the same as the address
> written in chosen node of DT. And xen image should be 2MiB aligned.
> 
>> 
>> Dom0 no longer aborts but the kernel is now stuck in the
>> __delay_loop function. It never returns. Ian pointed me
>> to an email from you on the mailing list 
>> (http://lists.xen.org/archives/html/xen-devel/2014-04/msg02620.html)
>> where you recommended to disable UART port in the kernel for OMAP5.
>> I am not doing that and was wondering if that could be causing the
>> kernel to hang ? Could you provide more details on what exactly
>> do I need to change in the kernel to disable UART port ?
> 
> The reason to disable UART3 hwmod is that the hypervisor won’t map
> the corresponding io memory address for dom0 because the UART is taken
> by itself. If the dom0 kernel doesn’t use hwmod (only use FDT), everything
> would work fine. Because Xen removes the corresponding UART node
> in the DT passed to dom0. However, since hwmod structure is used in
> OMAP kernel, the dom0 would still try to write the UART’s IO address
> space hard-coded in the hwmod structure, which could lead kernel panic.
> One of the ugly hack is to remove corresponding       structure in
> omap54xx_hwmod_ocp_ifs (arch/arm/mach-omap2/omap_hwmod_54xx_data.c).
> I should say I’m still not very familiar with omap kernel and have no
> idea whether there would be more elegant ways to solve this problem.

Just found that in the upstream kernel, there is no such issue. So there is
no need to disable hwmod manually in the kernel source if using the latest
upstream kernel :)

> 
>> 
>> I also wanted to ask you another question about how to get
>> linux printk's on xen console. Is it enough to pass
>> "console=hvc0"  to linux kernel or do I need to do something
>> more ?
> 
> To make the upstream kernel output info to xen-hvc, I did the following hacks:
> 
> diff --git a/drivers/tty/hvc/hvc_xen.c b/drivers/tty/hvc/hvc_xen.c
> index 2dc2831..931c72a 100644
> --- a/drivers/tty/hvc/hvc_xen.c
> +++ b/drivers/tty/hvc/hvc_xen.c
> @@ -630,7 +630,6 @@ void xen_raw_console_write(const char *str)
>        ssize_t len = strlen(str);
>        int rc = 0;
> 
> -       if (xen_domain()) {
>                rc = dom0_write_console(0, str, len);
> #ifdef CONFIG_X86
>                if (rc == -ENOSYS && xen_hvm_domain())
> @@ -642,7 +641,6 @@ outb_print:
>                for (i = 0; i < len; i++)
>                        outb(str[i], 0xe9);
> #endif
> -       }
> }
> 
> void xen_raw_printk(const char *fmt, ...)
> diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
> index a45b509..907922b 100644
> --- a/kernel/printk/printk.c
> +++ b/kernel/printk/printk.c
> @@ -45,6 +45,7 @@
> #include <linux/poll.h>
> #include <linux/irq_work.h>
> #include <linux/utsname.h>
> +#include <xen/hvc-console.h>
> 
> #include <asm/uaccess.h>
> 
> @@ -1571,6 +1572,8 @@ asmlinkage int vprintk_emit(int facility, int level,
>                }
>        }
> 
> +       xen_raw_console_write(text);
> +
>        if (level == -1)
>                level = default_message_loglevel;
> 
> Hopefully this is helpful.
> 
> Baozi
> 
>> 
>> Thanks for your help.
>> 
>> Best,
>> Ashish
>> 
>> -----Original Message-----
>> From: Chen Baozi [mailto:baozich@xxxxxxxxx] 
>> Sent: Saturday, April 26, 2014 8:04 PM
>> To: Kapania, Ashish
>> Cc: Julien Grall; Ian Campbell; xen-devel@xxxxxxxxxxxxx
>> Subject: Re: [Xen-devel] Problems running Xen 4.4 on OMAP5432 evm
>> 
>> Hi Ashish,
>> 
>> On Apr 26, 2014, at 0:56, Kapania, Ashish <akapania@xxxxxx> wrote:
>> 
>>> Hi Julien,
>>> 
>>> Here's the console output I get:
>>> 
>>> ## Booting kernel from Legacy Image at e0000000 ...
>>> Image Name:   Xen
>>> Image Type:   ARM Linux Kernel Image (uncompressed)
>>> Data Size:    656156 Bytes = 640.8 KiB
>>> Load Address: 80200000
>>> Entry Point:  80200000
>>> Verifying Checksum ... OK
>>> ## Flattened Device Tree blob at 825f0000
>>> Booting using the fdt blob at 0x825f0000
>>> Loading Kernel Image ... OK
>>> reserving fdt memory region: addr=825f0000 size=7000
>>> Using Device Tree in place at 825f0000, end 825f9fff
>>> 
>>> Starting kernel ...
>>> 
>>> - UART enabled -
>>> - CPU 00000000 booting -
>>> - Xen starting in Hyp mode -
>>> - Zero BSS -
>>> - Setting up control registers -
>>> - Turning on paging -
>>> - Ready -
>>> (XEN) Checking for initrd in /chosen
>>> (XEN) RAM: 0000000080000000 - 00000000feffffff
>>> (XEN)
>>> (XEN) MODULE[1]: 00000000825f0000 - 00000000825f7000
>>> (XEN) MODULE[2]: 0000000080300000 - 0000000080800000
>>> (XEN)  RESVD[0]: 00000000825f0000 - 00000000825f7000
>>> (XEN)
>>> (XEN) Command line: dom0_mem=128M sync_console console=dtuart 
>>> dtuart=serial2 noreboot
>>> (XEN) Placing Xen at 0x00000000fee00000-0x00000000ff000000
>>> (XEN) Xen heap: 00000000ee000000-00000000fe000000 (65536 pages)
>>> (XEN) Dom heap: 454656 pages
>>> (XEN) Domain heap initialised
>>> (XEN) Looking for UART console serial2 Xen 4.5-unstable
>>> (XEN) Xen version 4.5-unstable (arm-linux-gnueabihf-gcc (crosstool-NG 
>>> linaro-1.13.1-4.7-2013.03-20130313 - Linaro GCC 2013.03) 4.7.3 
>>> 20130226 (prerelease)) debug=y Thu Apr 24 10:04:50 PDT 2014
>>> (XEN) Latest ChangeSet: Mon Apr 14 15:14:47 2014 +0200 
>>> git:c82fbfe-dirty
>>> (XEN) Console output is synchronous.
>>> (XEN) Processor: 412fc0f2: "ARM Limited", variant: 0x2, part 0xc0f, 
>>> rev 0x2
>>> (XEN) 32-bit Execution:
>>> (XEN)   Processor Features: 00001131:00011011
>>> (XEN)     Instruction Sets: AArch32 Thumb Thumb-2 ThumbEE Jazelle
>>> (XEN)     Extensions: GenericTimer Security
>>> (XEN)   Debug Features: 02010555
>>> (XEN)   Auxiliary Features: 00000000
>>> (XEN)   Memory Model Features: 10201105 20000000 01240000 02102211
>>> (XEN)  ISA Features: 02101110 13112111 21232041 11112131 10011142 
>>> 00000000
>>> (XEN) Platform: TI OMAP5
>>> (XEN) Set AuxCoreBoot1 to 00000000fee0004c (0020004c)
>>> (XEN) Set AuxCoreBoot0 to 0x20
>>> (XEN) cpu node `/cpus/cpu@1`: #size-cells 1
>>> (XEN) DT missing boot CPU MPIDR[23:0]
>>> (XEN) Using only 1 CPU
>>> (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27
>>> (XEN) Using generic timer at 6144 KHz
>>> (XEN) GIC initialization:
>>> (XEN)         gic_dist_addr=0000000048211000
>>> (XEN)         gic_cpu_addr=0000000048212000
>>> (XEN)         gic_hyp_addr=0000000048214000
>>> (XEN)         gic_vcpu_addr=0000000048216000
>>> (XEN)         gic_maintenance_irq=25
>>> (XEN) GIC: 192 lines, 2 cpus, secure (IID 0000043b).
>>> (XEN) Using scheduler: SMP Credit Scheduler (credit)
>>> (XEN) Allocated console ring of 16 KiB.
>>> (XEN) VFP implementer 0x41 architecture 4 part 0x30 variant 0xf rev 
>>> 0x0
>>> (XEN) Brought up 1 CPUs
>>> (XEN) *** LOADING DOMAIN 0 ***
>>> (XEN) Populate P2M 0x88000000->0x90000000 (1:1 mapping for dom0)
>>> (XEN) Loading kernel from boot module 2
>>> (XEN) Loading zImage from 0000000080300000 to 
>>> 000000008fa00000-000000008feb1cc0
>>> (XEN) Loading dom0 DTB to 0x000000008f800000-0x000000008f806d8e
>>> (XEN) Scrubbing Free RAM: .................done.
>>> (XEN) Initial low memory virq threshold set at 0x4000 pages.
>>> (XEN) Std. Loglevel: All
>>> (XEN) Guest Loglevel: All
>>> (XEN) **********************************************
>>> (XEN) ******* WARNING: CONSOLE OUTPUT IS SYNCHRONOUS
>>> (XEN) ******* This option is intended to aid debugging of Xen by 
>>> ensuring
>>> (XEN) ******* that all output is synchronously delivered on the serial line.
>>> (XEN) ******* However it can introduce SIGNIFICANT latencies and 
>>> affect
>>> (XEN) ******* timekeeping. It is NOT recommended for production use!
>>> (XEN) **********************************************
>>> (XEN) 3... 2... 1... 
>>> (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch 
>>> input to Xen)
>>> (XEN) Freed 264kB init memory.
>>> 
>>> And the dom0 dump:
>>> 
>>> (XEN) 'q' pressed -> dumping domain info (now=0x12:5CD00FFD)
>>> (XEN) General information for domain 0:
>>> (XEN)     refcnt=3 dying=0 pause_count=0
>>> (XEN)     nr_pages=32768 xenheap_pages=2 shared_pages=0 paged_pages=0 
>>> dirty_cpus={} max_pages=4294967295
>>> (XEN)     handle=00000000-0000-0000-0000-000000000000 vm_assist=00000000
>>> (XEN) GICH_LRs (vcpu 0) mask=0
>>> (XEN)    VCPU_LR[0]=0
>>> (XEN)    VCPU_LR[1]=0
>>> (XEN)    VCPU_LR[2]=0
>>> (XEN)    VCPU_LR[3]=0
>>> (XEN) Rangesets belonging to domain 0:
>>> (XEN)     Interrupts { }
>>> (XEN)     I/O Memory { }
>>> (XEN) NODE affinity for domain 0: [0]
>>> (XEN) VCPU information and callbacks for domain 0:
>>> (XEN)     VCPU0: CPU0 [has=F] poll=0 upcall_pend = 00, upcall_mask = 01 
>>> dirty_cpus={} cpu_affinity={0-127}
>>> (XEN)     pause_count=0 pause_flags=0
>>> (XEN)     No periodic timer
>>> (XEN) Notifying guest 0:0 (virq 1, port 0)
>>> (XEN) 'd' pressed -> dumping registers
>>> (XEN)
>>> (XEN) *** Dumping CPU0 guest state (d0v0): ***
>>> (XEN) ----[ Xen-4.5-unstable  arm32  debug=y  Tainted:    C ]----
>>> (XEN) CPU:    0
>>> (XEN) PC:     0000000c
>>> (XEN) CPSR:   900001d7 MODE:32-bit Guest ABT
>>> (XEN)      R0: 80004000 R1: 00000c12 R2: 80008000 R3: 80004000
>>> (XEN)      R4: 80008000 R5: 00000000 R6: 0000000e R7: ffffffff
>>> (XEN)      R8: 8f800000 R9: 80000000 R10:90000000 R11:10201105 R12:8fa000a8
>>> (XEN) USR: SP: 00000000 LR: 00000000
>>> (XEN) SVC: SP: 00000000 LR: 8fa0036c SPSR:000001d3
>>> (XEN) ABT: SP: 00000000 LR: 0000000c SPSR:900001d7
>>> (XEN) UND: SP: 00000000 LR: 00000000 SPSR:00000000
>>> (XEN) IRQ: SP: 00000000 LR: 00000000 SPSR:00000000
>>> (XEN) FIQ: SP: 00000000 LR: 00000000 SPSR:00000000
>>> (XEN) FIQ: R8: 00000000 R9: 00000000 R10:00000000 R11:00000000 
>>> R12:00000000
>>> (XEN) 
>>> (XEN)      SCTLR: 00c50078
>>> (XEN)        TCR: 00000000
>>> (XEN)      TTBR0: 0000000000000000
>>> (XEN)      TTBR1: 0000000000000000
>>> (XEN)       IFAR: 0000000c, IFSR: 00000002
>>> (XEN)       DFAR: 80004000, DFSR: 00000002
>>> (XEN) 
>>> (XEN)   VTCR_EL2: 80003558
>>> (XEN)  VTTBR_EL2: 00010000825fe000
>>> (XEN)
>>> (XEN)  SCTLR_EL2: 30cd187f
>>> (XEN)    HCR_EL2: 0000000000282435
>>> (XEN)  TTBR0_EL2: 00000000feedf000
>>> (XEN) 
>>> (XEN)    ESR_EL2: 80000005
>>> (XEN)  HPFAR_EL2: 0000000000000000
>>> (XEN)      HDFAR: 80004000
>>> (XEN)      HIFAR: 0000000c
>>> (XEN)
>>> (XEN) Guest stack trace from sp=0:
>>> (XEN)   Failed to convert stack to physical address
>>> (XEN)
>>> 
>>> The above output is with development branch xen from Apr14. I get the same 
>>> output with xen v4.4 too.
>> 
>> I just tested the latest xen from git with a 3.11-rc3 dom0 kernel on my 
>> omap5432 uevm board. And seems all the problem I got is that dom0 was trying 
>> to access some addresses hardcoded in hwmod that the hypervisor didn't map 
>> according to the info it got from DT.
>> 
>> I'll have a test with latest upstream kernel these days and see what would 
>> happen.
>> 
>> Cheers,
>> 
>> Baozi
>> 
>>> 
>>> Best,
>>> Ashish
>>> 
>>> -----Original Message-----
>>> From: Julien Grall [mailto:julien.grall@xxxxxxxxxx]
>>> Sent: Friday, April 25, 2014 2:17 AM
>>> To: Kapania, Ashish; Ian Campbell
>>> Cc: xen-devel@xxxxxxxxxxxxx
>>> Subject: Re: [Xen-devel] Problems running Xen 4.4 on OMAP5432 evm
>>> 
>>> Hi Anish,
>>> 
>>> On 25/04/14 00:05, Kapania, Ashish wrote:
>>>> I apologize for causing any confusion. The xen.lds file I modified is 
>>>> generated and hence not tracked by git. I had forgotten about the change 
>>>> and running "git status"
>>>> was not telling me that I modified the file. Another point I wanted 
>>>> to make was that I always do a "make distclean" before rebuilding xen 
>>>> but the xen.lds file does not get regenerated. I had to manually delete it 
>>>> to force regeneration.
>>>> Maybe that's a bug ?
>>>> 
>>>> I tried the Ctrl-A * 3 + press 'd' sequence and found that dom0 is in ABT 
>>>> mode.
>>>> I tried debugging a little with JTAG (My JTAG setup does not work 
>>>> well in HYP Mode so I cannot debug xen with it) and found that the 
>>>> ABT happens in Linux/arch/arm/boot/compressed/head.S's __setup_mmu code.
>>>> The exact instruction that causes the abort is a store instruction 
>>>> that is attempting to write a page table entry to 0x80004000.
>>>> 
>>>> MMU stage 1 translation is disabled at this point so it looks like 
>>>> the Stage 2 translation is not mapping 0x80004000 address. Is there 
>>>> some step that I am missing that is suppose to tell xen to map the 
>>>> memory used by linux to store page tables ?
>>> 
>>> Can you post the console output of Xen and DOM0 (if you have some)? I'd 
>>> like to seems how Xen populates the RAM for DOM0.
>>> 
>>> Regards,
>>> 
>>> --
>>> Julien Grall
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@xxxxxxxxxxxxx
>>> http://lists.xen.org/xen-devel
>> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel