Re: arm (qemu -M virt) 64 bit xen running 32 bit guest problem

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Tue, 25 Aug 2020, Charles Chiou wrote:
> > (+ Stefano)
> >
> > On 25/08/2020 13:04, Charles Chiou wrote:
> > > Hello,
> >
> > Hello,
> >
> > > I'm having problem with virtual console when booting up 32 bit guests. I'm
> > using qemu virt machine, running xen 64 bit, DOM0 Linux is 64 bit. The 64 
> > bit
> > guests that I've tried (Linux, an RTOS) have no issues, but when I tried to 
> > run
> > 32 bit OS (tried linux, FreeRTOS from Galois, a bare-metal app), I get 
> > "Invalid
> > MFN 0x33a08" or similar messages when making HVC calls such as:
> > >
> > >          mov     r0, #0
> > >          mov     r1, #35
> > >          ldr     r2, =banner
> > >          mov     r12, #18
> > >          hvc     #0xea1<----
> > >
> > > and get this at the HVC call to XEN:
> > >
> > > (XEN) p2m.c:1919: d8v0: Invalid MFN 0x33a08
> > >
> > > Not sure if there's misconfiguration somewhere, or compile options, etc.
> > that caused this... Any pointer is greatly appreciated!
> > >
> > > Below is output of similar problem when booting up a 32 bit linux kernel:
> >
> > Looking at the code, this seems like an issue when trying to translate a 
> > guest
> > virtual address to a machine address.
> >
> > A few questions:
> >     - Which QEMU version are you using?
> >     - What's your Linux configuration? Are you using LPAE or short page 
> > tables?
> 
> I am using default ubuntu package on bionic:
> 
> $ qemu-system-aarch64 --version
> QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.29)
> Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
> 
> I just tried the stable-4.14 branch and this "Invalid MFN 0x..." doesn't 
> happen. It was on the master branch where I encountered this problem.
> However, I don't seem to get anything to call guest_printk() after making 
> hypervisor_console_io call even for 64-bit guest in this version (worked on 
> master). This is a separate question: could that be a XEN compile option 
> problem as I see "debug=n" from xen register dump:
> (XEN) ----[ Xen-4.14.1-pre  arm64  debug=n   Not tainted ]----

The call chain is:

xen/drivers/char/console.c:do_console_io
xen/drivers/char/console.c:guest_console_write
xen/drivers/char/console.c:guest_printk

You can enable debug through kconfig by doing "make menuconfig" under
xen/


> I'm ignoring 32-bit Linux for now and trying to boot bare metal apps that 
> calls hypervisor_console_io() at the very beginning of the boot-up. As I'm 
> trying to port embedded RTOS apps (both 32 and 64 bits) to run as guests.

Excellent. When doing that kind of work, I find the debug hypercalls very 
useful, see:

xen/arch/arm/traps.c:do_debug_trap

Once you enable DEBUG in the build, you can do

  hvc 0xfffd

In the guest for instance to print the program counter.



> Thank you!
> 
> 
> > I am also not sure what's the state of virtualization support in QEMU for 
> > 32-
> > bit guest. I have CCed Stefano who may have a better idea.
> >
> > Best regards,
> >
> > > / # xl -v create -c /share/misc/linux32.cfg Parsing config from
> > > /share/misc/linux32.cfg
> > > libxl: info: libxl_create.c:122:libxl__domain_build_info_setdefault:
> > > qemu-xen is unavailable, using qemu-xen-traditional instead: No such
> > > file or directory
> > > libxl: detail: libxl_create.c:623:libxl__domain_make: passthrough:
> > > disabled
> > > domainbuilder: detail: xc_dom_allocate: cmdline="rw root=/dev/ram
> > rdinit=/sbin/init earlyprintk=serial,ttyAMA0", features=""
> > > domainbuilder: detail: xc_dom_kernel_file: filename="/share/linux-domu-
> > 32bit/arch/arm/boot/zImage"
> > > domainbuilder: detail: xc_dom_malloc_filemap    : 5904 kB
> > > domainbuilder: detail: xc_dom_module_file: filename="/share/dom0-
> > rootfs.img.gz"
> > > domainbuilder: detail: xc_dom_malloc_filemap    : 13553 kB
> > > domainbuilder: detail: xc_dom_boot_xen_init: ver 4.15, caps
> > > xen-3.0-aarch64 xen-3.0-armv7l
> > > domainbuilder: detail: xc_dom_rambase_init: RAM starts at 40000
> > > domainbuilder: detail: xc_dom_parse_image: called
> > > domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary
> > loader ...
> > > domainbuilder: detail: loader probe failed
> > > domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM64)
> > loader ...
> > > domainbuilder: detail: xc_dom_probe_zimage64_kernel: kernel is not an
> > > arm64 Image
> > > domainbuilder: detail: loader probe failed
> > > domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM32)
> > loader ...
> > > domainbuilder: detail: loader probe OK
> > > domainbuilder: detail: xc_dom_parse_zimage32_kernel: called
> > > domainbuilder: detail: xc_dom_parse_zimage32_kernel: xen-3.0-armv7l:
> > > 0x40008000 -> 0x405cc200
> > > domainbuilder: detail: xc_dom_devicetree_mem: called
> > > domainbuilder: detail: xc_dom_mem_init: mem 128 MB, pages 0x8000
> > > pages, 4k each
> > > domainbuilder: detail: xc_dom_mem_init: 0x8000 pages
> > > domainbuilder: detail: xc_dom_boot_mem_init: called
> > > domainbuilder: detail: set_mode: guest xen-3.0-armv7l, address size 32
> > > domainbuilder: detail: populate_guest_memory: populating RAM @
> > > 0000000040000000-0000000048000000 (128MB)
> > > domainbuilder: detail: populate_one_size: populated 0x40/0x40 entries
> > > with shift 9
> > > domainbuilder: detail: meminit: placing boot modules at 0x472c2000
> > > domainbuilder: detail: meminit: ramdisk: 0x472c2000 -> 0x47fff000
> > > domainbuilder: detail: meminit: devicetree: 0x47fff000 -> 0x48000000
> > > domainbuilder: detail: xc_dom_build_image: called
> > > domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
> > 0x40008+0x5c5 at 0xffff93aea000
> > > domainbuilder: detail: xc_dom_alloc_segment:   kernel       : 0x40008000 
> > > ->
> > 0x405cd000  (pfn 0x40008 + 0x5c5 pages)
> > > domainbuilder: detail: xc_dom_load_zimage_kernel: called
> > > domainbuilder: detail: xc_dom_load_zimage_kernel: kernel seg
> > > 0x40008000-0x405cd000
> > > domainbuilder: detail: xc_dom_load_zimage_kernel: copy 6046208 bytes
> > > from blob 0xffff94dec000 to dst 0xffff93aea000
> > > domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
> > 0x472c2+0xd3d at 0xffff92dad000
> > > domainbuilder: detail: xc_dom_alloc_segment:   module0      : 0x472c2000 -
> > > 0x47fff000  (pfn 0x472c2 + 0xd3d pages)
> > > domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
> > 0x47fff+0x1 at 0xffff95847000
> > > domainbuilder: detail: xc_dom_alloc_segment:   devicetree   : 0x47fff000 
> > > ->
> > 0x48000000  (pfn 0x47fff + 0x1 pages)
> > > domainbuilder: detail: alloc_magic_pages: called
> > > domainbuilder: detail: xc_dom_build_image  : virt_alloc_end :
> > > 0x48000000
> > > domainbuilder: detail: xc_dom_build_image  : virt_pgtab_end : 0x0
> > > domainbuilder: detail: xc_dom_boot_image: called
> > > domainbuilder: detail: bootearly: doing nothing
> > > domainbuilder: detail: xc_dom_compat_check: supported guest type:
> > > xen-3.0-aarch64
> > > domainbuilder: detail: xc_dom_compat_check: supported guest type:
> > > xen-3.0-armv7l <= matches
> > > domainbuilder: detail: start_info_arm: called
> > > domainbuilder: detail: domain builder memory footprint
> > > domainbuilder: detail:    allocated
> > > domainbuilder: detail:       malloc             : 117 kB
> > > domainbuilder: detail:       anon mmap          : 0 bytes
> > > domainbuilder: detail:    mapped
> > > domainbuilder: detail:       file mmap          : 19457 kB
> > > domainbuilder: detail:       domU mmap          : 19468 kB
> > > domainbuilder: detail: vcpu_arm32: called
> > > domainbuilder: detail: Initial state CPSR 0x1d3 PC 0x40008000
> > > domainbuilder: detail: compat_gnttab_hvm_seed: d7: pfn=0x38000
> > > domainbuilder: detail: xc_dom_set_gnttab_entry: d7 gnt[0] -> d0
> > > 0x39000
> > > domainbuilder: detail: xc_dom_set_gnttab_entry: d7 gnt[1] -> d0
> > > 0x39001
> > > domainbuilder: detail: xc_dom_release: called
> > > (XEN) p2m.c:1919: d7v0: Invalid MFN 0x3b401
> > > (XEN) d7v0: vGICR: SGI: unhandled word write 0x000000ffffffff to
> > > ICACTIVER0
> > > (XEN) p2m.c:1919: d7v0: Invalid MFN 0x3b401
> > > (XEN) p2m.c:1919: d7v0: Invalid MFN 0x39281
> NOTE: This email (including attachments) contain Ambarella Proprietary and/or 
> Confidential Information and is intended solely for the use of the 
> individual(s) to whom it is addressed. Any unauthorized review, use, 
> disclosure, distribute, copy, or print is prohibited. If you are not an 
> intended recipient, please contact the sender by reply email and destroy all 
> copies of the original message. Thank you.
>