Re: [Xen-devel] [ARM:PATCH v1 1/1] Add Odroid-XU (Exynos5410) support

[Suriyan Ramasami <suriyan.r@xxxxxxxxx>]

Some more information ...
This time I started XEN with all the CPUs (4, all cpus uncommented in
the device tree).
dom0 started with no CPU restrictions => it is running with 4 VCPUs.
domU started with vcpus = 4.

root@dom0:~# xl vcpu-list
Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
Domain-0                             0     0    0   -b-      25.4  all
Domain-0                             0     1    2   -b-      22.6  all
Domain-0                             0     2    0   -b-      33.0  all
Domain-0                             0     3    1   r--      27.9  all
domU2                                3     0    3   r--     268.9  all
domU2                                3     1    -   --p       0.0  all
domU2                                3     2    -   --p       0.0  all
domU2                                3     3    -   --p       0.0  all

I have modified domU kernel so that it keeps looping and calling
arch_timer_get_cntfrq() till it returns a non zero value.
Note that domU VCPU 0 is on CPU 3. If I move it to run on CPU 0 (xl
vcpu-pin 3 0 0), then arch_timer_get_cntfrq() returns the correct
value and boot proceeds. Moving it to CPU 1, 2, or 3 does not get the
right value from arch_timer_get_cntfrq().

root@dom0:~# xl vcpu-pin 3 0 0
root@dom0:~# xl vcpu-list
Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
Domain-0                             0     0    2   -b-      28.6  all
Domain-0                             0     1    0   r--      24.5  all
Domain-0                             0     2    3   -b-      34.8  all
Domain-0                             0     3    1   -b-      29.7  all
domU2                                3     0    0   -b-     500.3  0
domU2                                3     1    2   -b-       3.2  all
domU2                                3     2    1   -b-       3.3  all
domU2                                3     3    2   -b-       3.1  all

This then boots up and I get domU console etc ...

So, why is any other CPU other than CPU0 not able to read the CNTFRQ
register? Is my CPU turn on code causing this issue, or is something
extra needed to be done to the other CPUs in the bring up code?

Thanks
- Suriyan



On Mon, Jul 28, 2014 at 1:19 PM, Suriyan Ramasami <suriyan.r@xxxxxxxxx> wrote:
> Hello Ian/Julien,
>     I did some more debugging and found this weirdness. It has me a
> little baffled.
>
> I added xen_raw_printk() as suggested by Ian in earlier posts as a
> mechanism for debugging domU (xen to be compiled with debug=y), and I
> have been slapping myself silly as to why I didn't try this before! It
> definitely gave me output that I could then hone in on!
>
> The times that domU was not booting up, it crashed with a "divide by
> zero" in the linux kernel because of the arch_timer_rate being 0. I
> would see the warning "Architected timer frequency not available" in
> the kernel log (xen output)
>
> This is indeed baffling! Especially as I am setting CNTFRQ to the the
> right value, and I know it is not 0!
>
> The times that domU boots up, it does read the CNTFRQ value correctly!
>
> This is the call trace that happens in drivers/clocksource/arm_arch_timer.c
>
> It comes into the function, and finds arch_timer_rate as 0.
> It tries to read clock_frequency property in the device tree. It is not set.
> It checks if cntbase is set. It is not set.
> It then calls arch_timer_get_cntfrq() to issue the mrc call.
> I have printed out the return value of arch_timer_get_cntfrq() and
> sometimes I get a 0 and sometimes I get 24000000.
>
> arch_timer_get_cntfrq is nothing but:
> static inline u32 arch_timer_get_cntfrq(void)
> {
>         u32 val;
>         asm volatile("mrc p15, 0, %0, c14, c0, 0" : "=r" (val));
>         return val;
> }
>
> Are the mrc calls trapped by XEN?
>
> Any pointers are appreciated!
> - Suriyan
>
>
> On Mon, Jul 28, 2014 at 11:06 AM, Suriyan Ramasami <suriyan.r@xxxxxxxxx> 
> wrote:
>> Hello Julien,
>>
>> On Mon, Jul 28, 2014 at 5:53 AM, Julien Grall <julien.grall@xxxxxxxxxx> 
>> wrote:
>>> Hi Suriyan,
>>>
>>> On 07/27/2014 07:12 PM, Suriyan Ramasami wrote:
>>>>>
>>>>>> I was wondering if dom0 is touching the physical CPUs by mistake (mainly
>>>>>> because we map some regions used to control them).
>>>>>> It might be worse to check that no cpufreq or power management is called
>>>>>> on DOM0 at any time.
>>>>>
>>>> I shall dig into this some more.
>>>> Thanks!
>>>
>>> I would give a try with:
>>>         - Xen: 2 CPUs
>>>         - DOM0: 1 VCPU
>>>         - Guest: 1 VCPU
>>>
>>> You can restrict the number of DOM0 VCPUs with dom0_max_vcpus=n where n
>>> is the number of VCPUs for DOM0.
>>>
>>> If it's not failing then the error is in Linux.
>>>
>> I did the above suggestion and I tried bringing up domU atleast 30
>> times. None of the attempts succeeded - domU gave no console output. I
>> also tried with a XEN compile with exynos5_specific_mapping not being
>> called for the Odroid-XU (don't know if this is of any relevance)
>>
>> Does this imply the problem is with Xen 4.5 dev branch?
>>
>> Relevant logs:
>> ------------ 8< ------------
>> XEN:
>> ...
>> (XEN) Command line: sync_console console=dtuart
>> dtuart=/serial@12C20000 earlyprintk=xen dom0_mem=512M dom0_max_vcpus=1
>> ...
>> (XEN) Bringing up CPU1
>> (XEN) exynos5.c:126: Power: 0 status: 0
>> (XEN) exynos5.c:130: Waiting for power status to change to 3
>> (XEN) exynos5.c:138: Power status changed to 3!
>> (XEN) CPU 1 booted.
>> (XEN) Brought up 2 CPUs
>>
>> DOM0:
>> ...
>> [    0.000000] Kernel command line: console=hvc0 earlyprintk debug
>> clk_ignore_unused root=/dev/mmcblk0p2 rootwait rw
>> ...
>> [    0.038352] Xen: initializing cpu0
>> [    0.038414] Setting up static identity map for 0x604940c8 - 0x60494114
>> [    0.039487] Brought up 1 CPUs
>> [    0.039496] SMP: Total of 1 processors activated.
>> [    0.039503] CPU: All CPU(s) started in SVC mode.
>>
>> root@dom0:~/Debug# xl vcpu-list
>> Name                                ID  VCPU   CPU State   Time(s) CPU 
>> Affinity
>> Domain-0                             0     0    0   r--     123.3  all
>> domU2                                2     0    1   r--     670.9  all
>>
>> And domU2's sample function from xenctx 2 is as follows: (domU2 is
>> linux mainline)
>> root@dom0:~/Debug# ./domUtrace.sh 2
>> 0xc0054c74 do_raw_spin_unlock
>> 0xc00085d0 gic_handle_irq
>> 0xc006c670 tick_periodic
>> 0xc00085d0 gic_handle_irq
>> 0xc006789c get_xtime_and_monotonic_and_sleep_offset
>> 0xc00085d0 gic_handle_irq
>> 0xc0054a90 do_raw_spin_lock
>> 0xc0021934 irq_exit
>> 0xc00085d0 gic_handle_irq
>> 0xc0054c60 do_raw_spin_unlock
>> 0xc00455e8 calc_global_load
>> 0xc00085d0 gic_handle_irq
>> 0xc006175c cpu_needs_another_gp
>> 0xc00085d0 gic_handle_irq
>> 0xc003b1b8 hrtimer_run_queues
>> 0xc00085d0 gic_handle_irq
>> ----------- 8< ----------------
>>
>> Thanks for all the pointers so far!
>>
>>> Regards,
>>>
>>> --
>>> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel