Re: [Minios-devel] Unikraft: _init_mem_clear_bootstrap() Fix

[Simon Kuenzer <simon.kuenzer@xxxxxxxxx>]

Hey Bruno,

thanks for sending me this. I had another look as well and I agree that 
_libxenplat_start should be at 0x0. Since we port some of the code 
(especially page table setup) from Mini-OS we should follow its expected 
boot layout, too:
0x0000 text section start
0x0000 _libxenplat_start
0x1000 shared info page
0x2000 hypercall page
0x3000 remaining text section
...

The first page at 0x0000 is erased during the boot by 
_init_mem_clear_bootstrap() (see plat/xen/x86/mm.c). In Mini-OS this 
page is used for a CoW implementation for the heap. _libxenplat_start() 
is setting up the stack and then calls _libxenplat_x86entry().
So, it is okay to erase _libxenplat_start() and reuse this page for 
something else. The code of _libxenplat_start() is not used anymore at 
later runtime. However, when the linker decides to place something else 
to 0x0, we have an issue (thanks for finding this!). ;-)
We have two options to fix the addresses of the symbols:
- .org note to force a position in the assembly code
- use a separate section as you suggested with .text.prologue

Since we use the second approach in also in arm (see .text.zimageboot in 
plat/xen/arm/link32.ld). I would prefer to do the same for x86. I would 
call the section .text.boot and also include the shared info and 
hypercall page to it.
Can you send me a final version of your patch? I will review it and push 
it upstream.
Thanks,

Simon


On 02.02.2018 18:17, Bruno Alvisio wrote:
> > On Feb 1, 2018, at 2:57 PM, Simon Kuenzer <simon.kuenzer@xxxxxxxxx 
> > <mailto:simon.kuenzer@xxxxxxxxx>> wrote:
> > Hi Bruno,
> Hi Simon,
>
> > thanks a lot for investigating this issue!
> > I had a similar issue with Xen on ARM where the zImage header was 
> > sometimes placed to a region that was out of range for the loader. I 
> > did not find the time yet to figure out why exactly this sometimes 
> > happens. I guess it has to do with the compiler flags -fdata-sections 
> > -ffunction-sections and --gc-sections for the linker. Any idea?
> I have been reading on some sites about this and looks like those flags 
> might make it work (they were mentioned in many discussions). 
> Unfortunately, I am not that familiar with all those flags but I will 
> try to understand with greaterdetail how they are used.
> > Could you send me your configuration (.config)? I wonder if I could 
> > reproduce the mis-placement that you saw; for now I couldn’t:
> Here is the .config file I am using:
>
> #
> # Automatically generated file; DO NOT EDIT.
> # Unikraft/0.2~8b94640-custom Configuration
> #
>
> #
> # Architecture Selection
> #
> ARCH_X86_64=y
> # ARCH_ARM_32 is not set
> # MARCH_NATIVE is not set
> MARCH_GENERIC=y
> # MARCH_NOCONA is not set
> # MARCH_CORE2 is not set
> # MARCH_COREI7 is not set
> # MARCH_COREI7AVX is not set
> # MARCH_COREI7AVXI is not set
> # MARCH_ATOM is not set
> # MARCH_K8 is not set
> # MARCH_K8SSE3 is not set
> # MARCH_AMDFAM10 is not set
> # MARCH_BTVER1 is not set
> # MARCH_BDVER1 is not set
> # MARCH_BDVER2 is not set
> # MARCH_BDVER3 is not set
> # MARCH_BTVER2 is not set
>
> #
> # Platform Support
> #
> PLAT_XEN=y
> XEN_DBGEMERGENCY=y
> # PLAT_KVM is not set
> # PLAT_LINUXU is not set
>
> #
> # Library Configuration
> #
> HAVE_BOOTENTRY=y
> # HAVE_LIBC is not set
> HAVE_SCHED=y
> LIBUKBOOT=y
> LIBUKBOOT_BANNER=y
> LIBUKBOOT_MAXNBARGS=60
> LIBUKBOOT_INITALLOC=y
> LIBUKDEBUG=y
> LIBUKDEBUG_PRINTK=y
> LIBUKDEBUG_PRINTD=y
> # LIBUKDEBUG_PRINTD_EXTRA is not set
> # LIBUKDEBUG_PRINTD_INFO is not set
> # LIBUKDEBUG_PRINTD_WARN is not set
> LIBUKDEBUG_PRINTD_ERR=y
> # LIBUKDEBUG_PRINTD_CRIT is not set
> LIBUKDEBUG_NOREDIR=y
> # LIBUKDEBUG_REDIR_PRINTD is not set
> # LIBUKDEBUG_REDIR_PRINTK is not set
> LIBUKDEBUG_ENABLE_ASSERT=y
> LIBUKARGPARSE=y
> LIBNOLIBC=y
> LIBUKALLOC=y
> LIBUKALLOC_IFPAGES=y
> # LIBUKALLOC_IFSTATS is not set
> LIBUKALLOCBBUDDY=y
> LIBUKSCHED=y
> LIBUKSCHEDCOOP=y
> # LIBFDT is not set
>
> #
> # Build Options
> #
> # OPTIMIZE_NONE is not set
> OPTIMIZE_PERF=y
> # OPTIMIZE_SIZE is not set
>
> #
> # Hint: Specify a CPU type to get most benefits from performance 
> optimization
> #
> # OPTIMIZE_DEADELIM is not set
> DEBUG_SYMBOLS=y
> # DEBUG_SYMBOLS_LVL1 is not set
> # DEBUG_SYMBOLS_LVL2 is not set
> DEBUG_SYMBOLS_LVL3=y
> OPTIMIZE_DBGFILE=y
> OPTIMIZE_SYMFILE=y
> OPTIMIZE_STRIP=y
> # RECORD_BUILDTIME is not set
> CROSS_COMPILE=""
>
> #
> # Application Options
> #
> APPHELLOWORLD_DEPENDENCIES=y
> APPHELLOWORLD_PRINTARGS=y
> UK_NAME=“helloworld"
>
> > nm build/helloworld_xen-x86_64.dbg | sort
> > 0000000000000000 T _libxenplat_start
> > 0000000000000000 T _text
> > 0000000000000008 a RSP_OFFSET
> > 0000000000000017 t stack_start
> > 00000000000000fc a KERNEL_CS_MASK
> > 0000000000001000 t _libxenplat_shared_info
> > 0000000000002000 t hypercall_page
> > 0000000000003000 t error_entry
> > 0000000000003069 t hypervisor_callback
> > 0000000000003078 t hypervisor_callback2
> > 00000000000030bb t error_exit
> > ...
> >
> > On 01.02.2018 07:35, Bruno Alvisio wrote:
> > > Hello all,
> > > Currently, the call to the function 
> > > unikraft/plat/x86/mm.c:_init_mem_clear_bootstrap() from 
> > > unikraft/plat/xen/x86/setup.c:_init_mem() causes unikraft to crash. I 
> > > investigated this issue and noticed that the VA address 
> > > of _libxenplat_start (kernel entry point) is located at address 
> > > 0x1000 instead of 0x0. (The .text section starts at address 0x0 
> > > though) See below:
> > > *balvisio@debianxen**:~/unikraft/apps/helloworld/build$ nm 
> > > helloworld_xen-x86_64.dbg | grep start | sort*
> > > *0000000000001000 T _libxenplat_start***
> > > *0000000000001017 t stack_start*
> > > *000000000000435b t thread_starter*
> > > *000000000000a0a0 t schedcoop_thread_start*
> > > *000000000000c00c d nonopt_start*
> > > *000000000000c01c D __bss_start*
> > > *0000000000057060 B HYPERVISOR_start_info***
> > > I forced the linker script to position _libxenplat_start() at 
> > > location 0x0 with the following patch and it seems to solve the problem:
> > > *diff --git a/plat/xen/x86/entry64.S b/plat/xen/x86/entry64.S*
> > > *index bdaa05a..94a5546 100644*
> > > *--- a/plat/xen/x86/entry64.S*
> > > *+++ b/plat/xen/x86/entry64.S*
> > > @@ -51,7 +51,7 @@ ELFNOTE(Xen, XEN_ELFNOTE_XEN_VERSION, .asciz "xen-3.0")
> > > .text
> > > .globl _libxenplat_start
> > > -
> > > +.section .text.prologue, "ax"
> > > _libxenplat_start:
> > > #else
> > > *diff --git a/plat/xen/x86/link64.ld b/plat/xen/x86/link64.ld*
> > > *index 2842a75..837c500 100644*
> > > *--- a/plat/xen/x86/link64.ld*
> > > *+++ b/plat/xen/x86/link64.ld*
> > > @@ -32,6 +32,7 @@ SECTIONS
> > > . = 0x0;
> > > _text = .;/* Text and read-only data */
> > > .text : {
> > > +*(.text.prologue)
> > > *(.text)
> > > *(.gnu.warning)
> > > } = 0x2000
> > > *diff --git a/plat/xen/x86/setup.c b/plat/xen/x86/setup.c*
> > > *index cf8bad2..18fcd24 100644*
> > > *--- a/plat/xen/x86/setup.c*
> > > *+++ b/plat/xen/x86/setup.c*
> > > @@ -156,7 +156,7 @@ static inline void _init_mem(void)
> > > uk_printd(DLVL_INFO, "max_pfn: %lx\n", max_pfn);
> > > _init_mem_build_pagetable(&start_pfn, &max_pfn);
> > > -//_init_mem_clear_bootstrap(); /* FIXME - stack or text screwed up? */
> > > +_init_mem_clear_bootstrap(); /* FIXME - stack or text screwed up? */
> > > //_init_mem_set_readonly(&_text, &_erodata); /* FIXME - shared info 
> > > ro? */
> > > /* Fill out mrd array */
> > > I am not sure if the problem can be solved by adding certain flag to 
> > > the linker so that it automatically places _libxenplat_start() 
> > > function at 0x0.
> > > Any comments on this would be greatly appreciated and if this looks 
> > > like a right solution I can send the PATCH to the list.
> > What happens actually to the other boot-critical symbols when you 
> > apply your patch (e.g., _libxenplat_shared_info, hypercall_page). For 
> > now the boot code expects them to be at 0x1000 and 0x2000.
> Looks like everything is shifted by 0x1000: (they end up at 0x2000 and 
> 0x3000 instead)
> balvisio@debianxen:~/unikraft/apps/helloworld/build$ nm 
> helloworld_xen-x86_64.dbg | grep shared_info | sort
> 0000000000002000 t _libxenplat_shared_info
> 0000000000057078 B HYPERVISOR_shared_info
>
> balvisio@debianxen:~/unikraft/apps/helloworld/build$ nm 
> helloworld_xen-x86_64.dbg | grep hypercall | sort
> 0000000000003000 t hypercall_page
>
> I send you the whole map in case you want to take a look:
>
> balvisio@debianxen:~/unikraft/apps/helloworld/build$ nm 
> helloworld_xen-x86_64.dbg | sort
> 0000000000000000 t do_hypervisor_callback
> 0000000000000000 T _text
> 0000000000000008 a RSP_OFFSET
> 00000000000000c0 t ukplat_lcpu_irqs_handle_pending
> 00000000000000fc a KERNEL_CS_MASK
> 0000000000000100 t ukplat_memregion_count
> 0000000000000110 t ukplat_memregion_get
> 00000000000002c0 t _libxenplat_x86entry
> 0000000000000420 t do_stack_walk
> 0000000000000470 t dump_mem.part.0
> 00000000000004f0 t dump_regs
> 00000000000005d0 t do_trap
> 0000000000000610 t do_divide_error
> 0000000000000630 t do_int3
> 0000000000000650 t do_overflow
> 0000000000000670 t do_bounds
> 0000000000000690 t do_invalid_op
> 00000000000006b0 t do_device_not_available
> 00000000000006d0 t do_coprocessor_segment_overrun
> 00000000000006f0 t do_invalid_TSS
> 0000000000000710 t do_segment_not_present
> 0000000000000730 t do_stack_segment
> 0000000000000750 t do_alignment_check
> 0000000000000770 t do_machine_check
> 0000000000000790 t stack_walk
> 00000000000007a0 t do_page_fault
> 00000000000008b0 t do_general_protection
> 0000000000000940 t do_debug
> 0000000000000980 t do_coprocessor_error
> 00000000000009b0 t simd_math_error
> 00000000000009d0 t do_simd_coprocessor_error
> 00000000000009f0 t do_spurious_interrupt_bug
> 0000000000000a00 t trap_init
> 0000000000000a30 t trap_fini
> 0000000000001000 T _libxenplat_start
> 0000000000001017 t stack_start
> 0000000000002000 t _libxenplat_shared_info
> 0000000000003000 t hypercall_page
> 0000000000004000 t error_entry
> 0000000000004069 t hypervisor_callback
> 0000000000004078 t hypervisor_callback2
> 00000000000040bb t error_exit
> 000000000000414a t restore_all_enable_events
> 00000000000041a8 t scrit
> 00000000000041b2 t restore_end
> 00000000000041d9 t ecrit
> 00000000000041d9 t hypervisor_prologue
> 00000000000041e1 t critical_region_fixup
> 0000000000004230 t failsafe_callback
> 0000000000004235 t coprocessor_error
> 0000000000004247 t simd_coprocessor_error
> 0000000000004259 t device_not_available
> 000000000000426b t debug
> 000000000000427d t int3
> 000000000000428f t overflow
> 00000000000042a1 t bounds
> 00000000000042b3 t invalid_op
> 00000000000042c5 t coprocessor_segment_overrun
> 00000000000042d7 t invalid_TSS
> 00000000000042e7 t segment_not_present
> 00000000000042f7 t stack_segment
> 0000000000004307 t general_protection
> 0000000000004317 t alignment_check
> 0000000000004327 t divide_error
> 0000000000004339 t spurious_interrupt_bug
> 000000000000434b t page_fault
> 000000000000435b t thread_starter
> 0000000000004369 t __arch_switch_threads
> 0000000000004390 t new_pt_frame
> 00000000000045a0 t _init_mem_build_pagetable
> 0000000000004850 t _init_mem_set_readonly
> 0000000000004a20 t _init_mem_clear_bootstrap
> 0000000000004ac0 t _init_mem_prepare
> 0000000000004b00 t arch_init_events
> 0000000000004b40 t arch_unbind_ports
> 0000000000004b50 t arch_fini_events
> 0000000000004b70 t ukplat_monotonic_clock
> 0000000000004c70 t timer_handler
> 0000000000004c90 t block_domain
> 0000000000004d20 t ukplat_time_init
> 0000000000004d50 t ukplat_time_fini
> 0000000000004d70 t arch_thread_init
> 0000000000004e20 t arch_run_idle_thread
> 0000000000004e30 t ukplat_lcpu_save_irqf
> 0000000000004e50 t ukplat_lcpu_restore_irqf
> 0000000000004e80 t ukplat_lcpu_irqs_disabled
> 0000000000004ea0 t ukplat_lcpu_halt_to
> 0000000000004f00 t ukplat_lcpu_halt
> 0000000000004f10 t hvconsole_input
> 0000000000004f20 t _libxenplat_prepare_console
> 0000000000004f60 t _libxenplat_init_console
> 0000000000005020 t ukplat_coutd
> 0000000000005060 t ukplat_coutk
> 00000000000051c0 t ukplat_cink
> 00000000000051d0 t ukplat_terminate
> 0000000000005210 t ukplat_suspend
> 0000000000005220 t default_handler
> 0000000000005240 t do_event
> 00000000000052c0 t bind_evtchn
> 0000000000005350 t unbind_evtchn
> 0000000000005420 t unbind_all_ports
> 00000000000054c0 t bind_virq
> 0000000000005520 t bind_pirq
> 0000000000005580 t init_events
> 00000000000055e0 t fini_events
> 0000000000005600 t evtchn_alloc_unbound
> 0000000000005670 t evtchn_bind_interdomain
> 00000000000056e0 t evtchn_get_peercontext
> 0000000000005740 t mask_evtchn
> 0000000000005770 t unmask_evtchn
> 00000000000057f0 t clear_evtchn
> 0000000000005820 t ukplat_thread_ctx_init
> 0000000000005830 t ukplat_thread_ctx_switch
> 0000000000005840 t ukplat_thread_ctx_current
> 0000000000005850 t ukplat_thread_ctx_run_idle
> 0000000000005860 t exit_thread
> 0000000000005900 t ukplat_entry
> 0000000000005aa0 t ukplat_entry_argp
> 0000000000005b30 t main_thread_func
> 0000000000005b50 t uk_vprintk
> 0000000000005b90 t uk_printk
> 0000000000005c50 t _uk_vprintd
> 0000000000005e90 t _uk_printd
> 0000000000006120 t _hxd_outf
> 0000000000006260 t _hxd
> 0000000000006770 t uk_hexdumpsn
> 0000000000006800 t uk_hexdumpf
> 0000000000006880 t uk_hexdumpk
> 00000000000068e0 t _uk_hexdumpd
> 0000000000006930 t uk_argnparse
> 0000000000006a90 t _nolibc_errno
> 0000000000006aa0 t vsnprintf
> 0000000000007720 t vsprintf
> 0000000000007740 t snprintf
> 00000000000077c0 t sprintf
> 0000000000007850 t vfprintf
> 00000000000078c0 t fprintf
> 0000000000007950 t vprintf
> 0000000000007970 t printf
> 0000000000007a10 t fflush
> 0000000000007a20 t strtoul
> 0000000000007c40 t strtoll
> 0000000000007f20 t strtoull
> 0000000000008150 t memcpy
> 0000000000008180 t memset
> 00000000000081b0 t memchr
> 00000000000081f0 t memmove
> 0000000000008250 t memcmp
> 00000000000082a0 t strlen
> 00000000000082d0 t strnlen
> 0000000000008320 t strncpy
> 00000000000083b0 t strcpy
> 00000000000083d0 t strchr
> 0000000000008420 t strncmp
> 0000000000008470 t permute_args
> 0000000000008530 t parse_long_options
> 00000000000087d0 t getopt_internal
> 0000000000008d50 t getopt
> 0000000000008d70 t getopt_long
> 0000000000008d80 t getopt_long_only
> 0000000000008d90 t malloc
> 0000000000008de0 t calloc
> 0000000000008e30 t realloc
> 0000000000008e80 t posix_memalign
> 0000000000008ee0 t memalign
> 0000000000008f20 t free
> 0000000000008f80 t uk_alloc_register
> 0000000000008fd0 t uk_alloc_get_default
> 0000000000008fe0 t uk_alloc_set_default
> 0000000000009040 t uk_malloc_ifpages
> 00000000000090e0 t uk_free_ifpages
> 0000000000009150 t uk_realloc_ifpages
> 0000000000009230 t uk_posix_memalign_ifpages
> 0000000000009330 t uk_calloc_compat
> 00000000000093a0 t uk_memalign_compat
> 0000000000009410 t map_free
> 0000000000009560 t bbuddy_addmem
> 0000000000009720 t bbuddy_palloc
> 00000000000099a0 t bbuddy_pfree
> 0000000000009bb0 t uk_allocbbuddy_init
> 0000000000009d60 t uk_sched_register
> 0000000000009db0 t uk_sched_get_default
> 0000000000009dc0 t uk_sched_set_default
> 0000000000009e20 t uk_sched_thread_create
> 0000000000009ee0 t uk_sched_thread_destroy
> 0000000000009f50 t uk_sched_sleep
> 0000000000009fc0 t uk_thread_block_millis
> 0000000000009ff0 t uk_thread_block
> 000000000000a010 t uk_thread_wake
> 000000000000a030 t idle_thread_fn
> 000000000000a0a0 t schedcoop_thread_start
> 000000000000a0e0 t schedcoop_run
> 000000000000a100 t schedcoop_schedule
> 000000000000a2d0 t schedcoop_thread_stop
> 000000000000a360 t uk_schedcoop_init
> 000000000000a4bd T _etext
> 000000000000a4bd T _rodata
> 000000000000a4c0 t main
> 000000000000a5a0 r pt_prot
> 000000000000a5c0 r __func__.3082
> 000000000000a740 r hex2ascii_data
> 000000000000a780 r _nolibc_ctype
> 000000000000a880 r __func__.2126
> 000000000000c000 D _data
> 000000000000c000 D _erodata
> 000000000000c000 d newline.1910
> 000000000000c004 d prevlvl.1911
> 000000000000c008 d nonopt_end
> 000000000000c00c d nonopt_start
> 000000000000c010 d optopt
> 000000000000c014 d optind
> 000000000000c018 d opterr
> 000000000000c01c D __bss_start
> 000000000000c01c D _edata
> 000000000000c020 d trap_table
> 000000000000c140 d stderr
> 000000000000c148 d stdout
> 000000000000c150 d stdin
> 000000000000c158 d place
> 000000000000c160 b _GLOBAL_OFFSET_TABLE_
> 000000000000c180 b cmdline
> 000000000000c580 b handling_pg_fault
> 000000000000c5a0 b mmu_updates.3503
> 000000000000e5c0 b mmu_updates.3486
> 00000000000105e0 b cpu0_pda
> 0000000000010600 b irqstack
> 0000000000030600 b port
> 0000000000030620 b shadow
> 0000000000030640 b console_ready
> 0000000000030644 b console_evtchn
> 0000000000030648 b console_ring
> 0000000000030660 b bound_ports
> 00000000000306e0 b ev_actions
> 00000000000366e0 b __errno
> 00000000000366e8 b uk_alloc_head
> 0000000000037000 B pt_base
> 0000000000037008 B _libxenplat_mrd_num
> 0000000000037020 B _libxenplat_mrd
> 0000000000037050 B optarg
> 0000000000037060 B _libxenplat_bootstack
> 0000000000057060 B HYPERVISOR_start_info
> 0000000000057068 B in_callback
> 000000000005706c B optreset
> 0000000000057070 B uk_sched_head
> 0000000000057078 B HYPERVISOR_shared_info
> 0000000000057080 B mfn_zero
> 0000000000057088 B phys_to_machine_mapping
> 0000000000057090 B _end
> 0000000080000000 a NMI_MASK
>
>
> > > Cheers >
> > > Bruno
> > Cheers,
> >
> > Simon
_______________________________________________
Minios-devel mailing list
Minios-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/minios-devel