Re: Keystone Issue

[CodeWiz2280 <codewiz2280@xxxxxxxxx>]

Hi Julien,

As requested please see log below from the eval board booting dom0, some notes are as follows:

1. The offset that gets applied to the 32-bit address to translate it to 36-bits is 0x7_8000_0000

2. Uboot has been setup to not change the address of the memory in the device tree prior to launching xen, otherwise it would automatically offset it and replace it with a 36-bit address and xen would immediately panic at the 36-bit address for a 32-bit processor.

3. The RAM starting address placed in the device tree is 0x8000_0000, which gets carved up by xen and replaced with 0xA000_0000 prior to booting dom0..  I had to put in test code to have the kernel offset the 0xA000_0000 32-bit starting address to the 36-bit address needed before the kernel will attempt to switch.  If it stays 32-bit then it will not switch over the address space.  Note that without xen in play uboot would normally replace the address in the device tree with the 36-bit one.

4. The dom0 kernel will boot from xen if the early_paging_init switch step is skipped, and the low mem stays in 32-bit....but there is a problem with the peripherals so this is not an acceptable solution.

It seems that either the kernel would need some API to tell xen that there is going to be a change in the memory its using prior to call early_paging_init(), or Xen would need to add the additional 36-bit addresses during the memory bank allocation step....but recognize that they are not actually different physical memory but just aliased to a different address.

Thanks,

Dave

 Xen 4.14-unstable
(XEN) Xen version 4.14-unstable (arm-linux-gnueabihf-gcc (Linaro GCC 4.9-2015.05) 4.9.3 20150413 (prerelease)) debug=y  Mon Jun  1 10:22:11 EDT 2020
(XEN) Latest ChangeSet:
(XEN) build-id: 30ae91a06c71a885cfba2788965144999a864614
(XEN) Console output is synchronous.
(XEN) Processor: 412fc0f4: "ARM Limited", variant: 0x2, part 0xc0f, rev 0x4
(XEN) 32-bit Execution:
(XEN)   Processor Features: 00001131:00011011
(XEN)     Instruction Sets: AArch32 A32 Thumb Thumb-2 ThumbEE Jazelle
(XEN)     Extensions: GenericTimer Security
(XEN)   Debug Features: 02010555
(XEN)   Auxiliary Features: 00000000
(XEN)   Memory Model Features: 10201105 20000000 01240000 02102211
(XEN)  ISA Features: 02101110 13112111 21232041 11112131 10011142 00000000
(XEN) Using SMC Calling Convention v1.0
(XEN) Using PSCI v0.1
(XEN) SMP: Allowing 4 CPUs
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 208333 KHz
(XEN) GICv2 initialization:
(XEN)         gic_dist_addr=0000000002561000
(XEN)         gic_cpu_addr=0000000002562000
(XEN)         gic_hyp_addr=0000000002564000
(XEN)         gic_vcpu_addr=0000000002566000
(XEN)         gic_maintenance_irq=25
(XEN) Using the new VGIC implementation.
(XEN) GICv2: 512 lines, 4 cpus, secure (IID 0200143b).
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN)  load_precision_shift: 18
(XEN)  load_window_shift: 30
(XEN)  underload_balance_tolerance: 0
(XEN)  overload_balance_tolerance: -3
(XEN)  runqueues arrangement: socket
(XEN)  cap enforcement granularity: 10ms
(XEN) load tracking window length 1073741824 ns
(XEN) Allocated console ring of 32 KiB.
(XEN) VFP implementer 0x41 architecture 4 part 0x30 variant 0xf rev 0x0
(XEN) CPU0: Guest atomics will try 2 times before pausing the domain
(XEN) Bringing up CPU1
(XEN) CPU1: Guest atomics will try 1 times before pausing the domain
(XEN) CPU 1 booted.
(XEN) Bringing up CPU2
(XEN) CPU2: Guest atomics will try 1 times before pausing the domain
(XEN) CPU 2 booted.
(XEN) Bringing up CPU3
(XEN) CPU3: Guest atomics will try 1 times before pausing the domain
(XEN) CPU 3 booted.
(XEN) Brought up 4 CPUs
(XEN) I/O virtualisation disabled
(XEN) P2M: 40-bit IPA
(XEN) P2M: 3 levels with order-1 root, VTCR 0x80003558
(XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
(XEN) Adding cpu 0 to runqueue 0
(XEN)  First cpu on runqueue, activating
(XEN) Adding cpu 1 to runqueue 0
(XEN) Adding cpu 2 to runqueue 0
(XEN) Adding cpu 3 to runqueue 0
(XEN) alternatives: Patching with alt table 002c2530 -> 002c2578
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading d0 kernel from boot module @ 0000000083000000
(XEN) Loading ramdisk from boot module @ 0000000088000000
(XEN) Allocating 1:1 mappings totalling 1024MB for dom0:
(XEN) BANK[0] 0x000000a0000000-0x000000e0000000 (1024MB)
(XEN) Grant table range: 0x00000082000000-0x00000082040000
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) Loading zImage from 0000000083000000 to 00000000a7a00000-00000000a7f36100
(XEN) Loading d0 initrd from 0000000088000000 to 0x00000000a8200000-0x00000000abe00000
(XEN) Loading d0 DTB to 0x00000000a8000000-0x00000000a8007872
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) ***************************************************
(XEN) WARNING: CONSOLE OUTPUT IS SYNCHRONOUS
(XEN) This option is intended to aid debugging of Xen by ensuring
(XEN) that all output is synchronously delivered on the serial line.
(XEN) However it can introduce SIGNIFICANT latencies and affect
(XEN) timekeeping. It is NOT recommended for production use!
(XEN) ***************************************************
(XEN) WARNING: SILO mode is not enabled.
(XEN) It has implications on the security of the system,
(XEN) unless the communications have been forbidden between
(XEN) untrusted domains.
(XEN) ***************************************************
(XEN) 3... 2... 1...
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 328kB init memory.
(XEN) DOM0: [    0.000000] Booting Linux on physical CPU 0x0
(XEN) DOM0: [    0.000000] Linux version 4.19.59-g5f8c1c6121 (gcc version 8.3.0 (GNU Toolchain for the A-profile A
(XEN) DOM0: rchitecture 8.3-2019.03 (arm-rel-8.36))) #52 SMP Mon Jun 1 12:13:51 EDT 2020
(XEN) DOM0: [    0.000000] CPU: ARMv7 Processor [412fc0f4] revision 4 (ARMv7), cr=30c5387d
(XEN) DOM0: [    0.000000] CPU: div instructions available: patching division code
(XEN) DOM0: [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
(XEN) DOM0: [    0.000000] OF: fdt: Machine model: Texas Instruments Keystone 2 Edison EVM
(XEN) DOM0: [    0.000000] bootconsole [earlycon0] enabled
(XEN) DOM0: [    0.000000] debug: ignoring loglevel setting.
(XEN) DOM0: [    0.000000] Memory policy: Data cache writealloc
(XEN) DOM0: [    0.000000] test : mem_start from dtb = 0xa0000000
(XEN) DOM0: [    0.000000] test : force the mem_start = 0x800000000
(XEN) DOM0: [    0.000000] test : note KEYSTONE_LOW_PHYS_START = 80000000, KEYSTONE_LOW_PHYS_END = ffffffff
(XEN) DOM0: [    0.000000] test : note KEYSTONE_HIGH_PHYS_START = 800000000, KEYSTONE_HIGH_PHYS_END = bffffffff
(XEN) DOM0: [    0.000000] test : Switch physical address space to 0x820000000 (0xa0000000 + 0x780000000)
(XEN) DOM0: [    0.000000] test : inside of early_paging_init()
(XEN) traps.c:1980:d0v0 HSR=0x80000086 pc=0xa020010c gva=0xa020010c gpa=0x0000082000310c
(XEN) traps.c:1980:d0v0 HSR=0x80000086 pc=0xffff000c gva=0xffff000c gpa=0x0000082000700c
(XEN) traps.c:1980:d0v0 HSR=0x80000086 pc=0xffff000c gva=0xffff000c gpa=0x0000082000700c
... last line loops indefinitely

On Mon, Jun 1, 2020 at 11:21 AM CodeWiz2280 <codewiz2280@xxxxxxxxx> wrote:

Hi Julien,

Thank you for your response.  I will try and post a log for you.  I have been switching back and forth between configurations and need to take a new one.

The board has 4GB of memory. Uboot places the kernel/initramfs/dtb in the 0x8000_0000 region but then the kernel switches its code/data over to a 0x8_0000_0000 range via the pv-fixup-asm.S assembly code called from early_paging_init in arch/arm/mm/mmu.c.  That code is exclusive to the keystone in the 4.19 kernel when "CONFIG_ARM_PV_FIXUP" and "ARM_LPAE" are enabled in the kernel .  The upper 2GB of memory is above 0xFFFF_FFFF so LPAE is required. 

/proc/iomem looks like this without running xen after the switch and the kernel boots:

80000000 - 9fffffff : System RAM (boot alias)

c8000000 - ffffffff : System RAM (boot alias)

800000000 - 1fffffff : System RAM

    800008000-800dfffff : Kernel Code

    801000000-80108ab3f : Kernel data

848000000-8ffffffff : System RAM

I was able to duplicate this issue with a build of your latest "master" repository from this morning.

On Mon, Jun 1, 2020 at 9:29 AM Julien Grall <julien@xxxxxxx> wrote:

Hello,

I have a few questions in order to understand a bit more your problem.

On 01/06/2020 13:38, CodeWiz2280 wrote:
> Hello, I am using a Texas Instruments K2E Keystone Eval board with Linux
> 4.19.59.  It has a 32-bit ARM Cortex A15 processor. There is keystone
> specific code in the kernel in arch/arm/mm/pv-fixup-asm.s that executes
> during early_paging_init for LPAE support.  This causes the kernel to
> switch its running 32-bit address space to a 36-bit address space and
> the hypervisor traps repeatedly and stops it from booting.

Without any log it is going to be difficult to help. Could you post the
hypervisor log when debug is enabled?

>  I suspect
> this is because Xen only allowed for the original 32-bit memory range
> specified by the dom0 device tree.

How much RAM did you give to your Dom0?

> The 36-bit LPAE address is a fixed
> offset from the 32-bit address and is not physically different memory.

I am not sure to understand this. Are you suggesting that the kernel is
trying to relocate itself in a different part of the physical memory?

Can you provide more details on the fixed offset?

> 
> Can you suggest any way to get through this problem? I am using the
> master branch of xen from earlier this year. 

Can you provide the exact baseline your are using? Did make any changes
on top?

> Any help is greatly
> appreciated.
Best regards,

--
Julien Grall