Re: Porting Xen to Jetson Nano

[srini@xxxxxxxxxx]

Hi Julien,

On 2020-07-24 17:25, Julien Grall wrote:
> On 24/07/2020 16:01, Srinivas Bangalore wrote:
>
> I struggled to find your comment inline as your e-mail client doesn't
> quote my answer. Please configure your e-mail client to use some form
> of quoting (the usual is '>').
I have switched to a web-based email client, so I hope this is better 
now.
> > (XEN) Freed 296kB init memory.
> > (XEN) dom0 IPA 0x0000000088080000
> > (XEN) P2M @ 0000000803fc3d40 mfn:0x17f0f5
> > (XEN) 0TH[0x0] = 0x004000017f0f377f
> > (XEN) 1ST[0x2] = 0x02c00000800006fd
> > (XEN) Mem access check
> > (XEN) dom0 IPA 0x0000000088080000
> > (XEN) P2M @ 0000000803fc3d40 mfn:0x17f0f5
> > (XEN) 0TH[0x0] = 0x004000017f0f377f
> > (XEN) 1ST[0x2] = 0x02c00000800006fd
> > (XEN) Mem access check
> The instruction abort issue looks normal as the mapping is marked as
> non-executable.
>
> Looking at the rest of bits, bits 55:58 indicates the type of mapping
> used. The value suggest the mapping has been considered to be used
> p2m_mmio_direct_c (RW cacheable MMIO). This looks wrong to me because
> RAM should be mapped using p2m_ram_rw.
>
> Looking at your DT, it looks like the region is marked as reserved. On
> Xen 4.8, reserved-memory region are not correctly handled (IIRC this
> was only fixed in Xen 4.13). This should be possible to confirm by
> enable CONFIG_DEVICE_TREE_DEBUG in your .config.
>
> The option will debug more information about the mapping to dom0 on
> your console.
>
> However, given you are using an old release, you are at risk at keep
> finding bugs that have been resolved in more recent releases. It would
> probably better if you switch to Xen 4.14 and report any bug you can
> find there.
Ok. I applied to patch series to 4.14. Enabled EARLY_PRINTK, 
CONFIG_DEBUG and DEVICE_TREE_DEBUG.
Here's the log...

## Flattened Device Tree blob at e3500000
   Booting using the fdt blob at 0xe3500000
   reserving fdt memory region: addr=80000000 size=20000
   reserving fdt memory region: addr=e3500000 size=35000
   Loading Device Tree to 00000000fc7f8000, end 00000000fc82ffff ... OK

Starting kernel ...

- UART enabled -
- Boot CPU booting -
- Current EL 00000008 -
- Initialize CPU -
- Turning on paging -
- Zero BSS -
- Ready -
(XEN) Invalid size for reg
(XEN) fdt: node `reserved-memory': parsing failed
(XEN)
(XEN) MODULE[0]: 00000000e0000000 - 00000000e014b0c8 Xen
(XEN) MODULE[1]: 00000000fc7f8000 - 00000000fc82d000 Device Tree
(XEN)  RESVD[0]: 0000000080000000 - 0000000080020000
(XEN)  RESVD[1]: 00000000e3500000 - 00000000e3535000
(XEN)  RESVD[2]: 00000000fc7f8000 - 00000000fc82d000
(XEN)  RESVD[3]: 0000000040001000 - 000000004003ffff
(XEN)  RESVD[4]: 00000000b0000000 - 00000000b01fffff
(XEN)
(XEN)
(XEN) Command line: console=dtuart sync_console dom0_mem=128M 
loglvl=debug guest_loglvl=debug console_to_ring
(XEN) Xen BUG at page_alloc.c:398
(XEN) ----[ Xen-4.14.0  arm64  debug=y   Not tainted ]----
(XEN) CPU:    0
(XEN) PC:     00000000002b7b90 alloc_boot_pages+0x38/0x9c
(XEN) LR:     00000000002cda04
(XEN) SP:     0000000000307d40
(XEN) CPSR:   a00003c9 MODE:64-bit EL2h (Hypervisor, handler)
(XEN)      X0: 000fc80000002000  X1: 0000000000002000  X2: 
0000000000000000
(XEN)      X3: 000fffffffffffff  X4: ffffffffffffffff  X5: 
0000000000000000
(XEN)      X6: 0000000000307df0  X7: 0000000000000003  X8: 
0000000000000008
(XEN)      X9: fffffffffffffffb X10: 0101010101010101 X11: 
0000000000000007
(XEN)     X12: 0000000000000004 X13: ffffffffffffffff X14: 
ffffffffff000000
(XEN)     X15: ffffffffffffffff X16: 0000000000000000 X17: 
0000000000000000
(XEN)     X18: 00000000fc834dd0 X19: 00000000002b5000 X20: 
00000000fc7f8000
(XEN)     X21: 00000000fc7f8000 X22: 0000000000000000 X23: 
fc80000000000038
(XEN)     X24: 00000000fed9de28 X25: ffffffffffffffff X26: 
fc80000002000000
(XEN)     X27: 0000000002000000 X28: 0000000000000000  FP: 
0000000000307d40
(XEN)
(XEN)   VTCR_EL2: 80000000
(XEN)  VTTBR_EL2: 0000000000000000
(XEN)
(XEN)  SCTLR_EL2: 30cd183d
(XEN)    HCR_EL2: 0000000000000038
(XEN)  TTBR0_EL2: 00000000e0145000
(XEN)
(XEN)    ESR_EL2: f2000001
(XEN)  HPFAR_EL2: 0000000000000000
(XEN)    FAR_EL2: 0000000000000000
(XEN)
(XEN) Xen stack trace from sp=0000000000307d40:
(XEN)    0000000000307df0 00000000002cf114 0000000000000000 
0000000000307d68
(XEN)    00000000fc7f8000 00000000002ceeb0 0000000000400000 
00676e69725f6f74
(XEN)    ffffffffffffffff 0000000000000000 0000000000000000 
0000000000307df0
(XEN)    0000000000307df0 00000000002cef58 000000003fffffff 
00000000fc7f8000
(XEN)    00000000fc7f8000 000fc80000002000 0000000000400000 
0080000000000000
(XEN)    0000000000000000 000000000003ffff 00000000fc831170 
00000000002001b8
(XEN)    00000000e0000000 00000000dfe00000 00000000fc7f8000 
0000000000000000
(XEN)    0000000000400000 00000000fed9de28 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000400 0000000000000000 
0000000000035000
(XEN)    00000000fc7f8000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000300000000 0000000000000000 
00000040ffffffff
(XEN)    00000000ffffffff 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN) Xen call trace:
(XEN)    [<00000000002b7b90>] alloc_boot_pages+0x38/0x9c (PC)
(XEN)    [<00000000002cda04>] setup_frametable_mappings+0xb4/0x310 (LR)
(XEN)    [<00000000002cf114>] start_xen+0x3a0/0xc48
(XEN)    [<00000000002001b8>] arm64/head.o#primary_switched+0x10/0x30
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Xen BUG at page_alloc.c:398
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...

There seems to be a problem with the DT in the 'reserved-memory' node.  
I commented out the fb0-carveout, fb1-carveout sections, recompiled and 
tried to boot again. This time the log shows the device tree messages 
(see attached log file), but Xen fails at this point...
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) Create hypervisor node
(XEN) Create PSCI node
(XEN) Create cpus node
(XEN) Create cpu@0 (logical CPUID: 0) node
(XEN) Create cpu@1 (logical CPUID: 1) node
(XEN) Create cpu@2 (logical CPUID: 2) node
(XEN) Create cpu@3 (logical CPUID: 3) node
(XEN) Create memory node (reg size 4, nr cells 4)
(XEN)   Bank 0: 0xe8000000->0xf0000000
(XEN) Create memory node (reg size 4, nr cells 8)
(XEN)   Bank 0: 0x40001000->0x40040000
(XEN)   Bank 1: 0xb0000000->0xb0200000
(XEN) Loading zImage from 00000000e1000000 to 
00000000e8080000-00000000ea23c808
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Unable to copy the kernel in the hwdom memory
(XEN) ****************************************
(XEN)

Device tree and log file attached. Is there an issue with the DT? Any 
pointers on where I should be looking next?
Thanks for your help.

Regards,
Srini

Attachment: jetson-nano-b00.dts
Description: Text document

Attachment: jetson-nano-log-with fdt-fix.txt
Description: Text document