Re: [Minios-devel] Some considerations of ARM Unikraft supports

[Wei Chen <Wei.Chen@xxxxxxx>]

Hi Julien,

Thanks for your comments!
Replies inline.

> -----Original Message-----
> From: Julien Grall [mailto:julien.grall@xxxxxxxxxx]
> Sent: 2018年2月2日 18:43
> To: Wei Chen <Wei.Chen@xxxxxxx>; Simon Kuenzer <simon.kuenzer@xxxxxxxxx>
> Cc: Felipe Huici <Felipe.Huici@xxxxxxxxx>; Kaly Xin <Kaly.Xin@xxxxxxx>; Shijie
> Huang <Shijie.Huang@xxxxxxx>; Florian Schmidt <Florian.Schmidt@xxxxxxxxx>;
> Costin Lupu <costin.lup@xxxxxxxxx>; nd <nd@xxxxxxx>; minios-
> devel@xxxxxxxxxxxxx
> Subject: Re: [Minios-devel] Some considerations of ARM Unikraft supports
> 
> Hi,
> 
> On 02/02/18 09:10, Wei Chen wrote:
> > This week I am trying to boot Unikraft on ARM64/KVM platform. In this
> progress I have
> > got some considerations and written a simple proposal:
> >
> > My first target is to enable Unikraft on ARM64+Kvm, so this proposal would
> focus on ARM64+Kvm.
> > But the goal of ARM support is to enable Unikraft on ARM32/ARM64 based
> hypervisors (ARM32/64 Kvm,
> > ARM64 Xen and etc). So we have to consider to keep current multi-arch
> framework and reuse common
> > code like virtual drivers for ARM32/ARM64.
> >
> > 1. Modify the folders for multi-architectures
> >      1.1. Add arm64 folder to unikraft/arch:
> >                  unikraft----arch----arm
> >                                |-----x86_64
> >                                |-----arm64 <-- New
> >
> >           Above folders contains architecture specified Makefile, Config,
> Compiler flags and some
> >           code. In most cases, these files are exclusive. So we'd better
> keep each arcitecture in
> >           a standalone floder. This also can avoid doing to much changes to
> Unikraft Makefile.
> >
> >           If we add arm64 to unikraft/arch/arm, we have to do more ARCH
> comparasion in Makefile:
> >                  unikraft----arch----arm----arm32
> >                            |      |-----arm64 <-- New
> >                       |
> >                            |-----x86_64
> >           Before:$(UK_BASE)/arch/$(ARCH)/Makefile.uk.
> >           After:$(UK_BASE)/arch/arm/$(ARCH)/Makefile.uk
> >           This change is complex, so we'd better to add arm64 folder to
> unikraft/arch.
> 
> Except the assembly code, most of the C code should be very similar
> between ARM64 and ARM32. So it might make more sense to have a directory
> arch/arm with sub-folder arm32 and arm64.
>

This is one option I had considered. But this will add a new variable (VENDOR) 
to
make scripts. e.g. :$(UK_BASE)/arch/$(VENDOR)/$(ARCH)/Makefile.uk
And currently, only architecture dependent code will be placed in $(ARCH) 
folder.
For example, in arm folder, there are some files for arm32 math library. These
files can only be used for arm32.

If some C codes are very similar between arm32 and arm64, I think this code 
would
be very similar between arm and x86 too. We can place these codes in 
Unikraft/lib.

Above 2 options would affect the common framework, so I still want to get some
Comments from Simon.

> >
> >      1.2. Add arm64 to unikraft/include/uk/arch
> >
> >      1.3. Add arm64 kvm platform code to unikraft/plat/kvm/arm, and use
> Makefile to select
> >           objects for correct architecutre:
> >
> >           ifeq ($(ARCH_X86_64),y)
> >              LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/x86/entry64.S
> >              LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/x86/cpu_x86_64.c
> >           else ifeq ($(ARCH_ARM_64),y)
> >              LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/entry64.S
> >              LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/cpu_arm64.c
> >           else ifeq ($(ARCH_ARM_64),y)
> >              LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/entry.S
> >              LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/cpu_arm.c
> >           endif
> >
> >      1.4. Add a "drivers" folder to unikraft/
> >           This because we may have some virtual device drivers can be shared
> among platforms.
> >           For example, we can reuse virtual uart, timer and gic drivers from
> arm32/arm64 Kvm/xen.
> >
> > 2. Bootloader
> >      2.1. Because of the BIOS, x86 is using multiboot to load kernel on
> Linux-KVM QEMU. But on ARM platforms,
> >           we can skip the EFI and boot from the Virtual Machine's RAM base
> address. So we can place _libkvmplat_entry
> >           to the CPU's reset entry by link script. On ARM64 platform, the
> default virtual machine CPU model is cortex A15.
> 
> Cortex A15 does not support 64-bit. So how come it is the default
> virtual machine CPU model for ARM64?
> 

From the code, if we don't specify any cpumodel, the mach-virt's default
cpumodel will be set to "cortex-a15". But you'are right, if we use cortex-15
by default, we can run any 64-bit image. Here is my mistake. We have to set
correct cpumodel (cortex-a53/a57 or host) in command line to make 64-bit image
work. But the mach-virt is still using the a15memmap and a15irqmap.


> But likely, you want to expose the same MIDR as the underlying CPU. So
> if an errata has to be implemented in Unikraft, it will be able to know it.
> 

Exposing the underlying CPU's MIDR to guest is depending on the hypervisors.
For Unikraft itself, it doesn't know whether this MIDR is the same as the 
underlying
CPU or not. And actually, no matter what cpumodel the hypervisor is emulating, 
the
code is running on the physical CPU directly. We don't emulate the CPU 
instructions.
If we run Unikraft on a corext-a53 host CPU, we can compile this image with gcc 
flags
like fix-a53-error.

> >
> >           plat/kvm/arm/link64.ld:
> >           ENTRY(_libkvmplat_entry)
> >           SECTIONS {
> >               . = 0x40000000;
> >
> >               /* Code */
> >               _stext = .;
> >
> >               .text :
> >               {
> >                   *(.text)
> >                   *(.text.*)
> >               }
> >
> >               _etext = .;
> >               ...
> >           }
> >
> >      2.2. Use the fixed physical addresses of PL011 uart, timer and GIC. So
> we can skip the device tree parse.
> 
> What does promise you the PL011, timer, GIC will always be at the same
> address?

My original idea was that we selected a fixed machine (mach-virt) for Unikraft 
to run.
In this case, the memory map is fixed.

> Or do you expect the user to hack unikraft build system to set
> the address?
> 

For my opinion, Yes. Why should we need to parse the device tree to increase 
our boot
time and footprint?

> At least from Xen PoV, the memory layout is not part of the ABI and a
> guest should rely on the DT for getting the correct addresses.
> 

I understand your concern. It's not a part of the ABI. So the addresses can be 
changed
for different boards.

I think we must do a tradeoff between flexibility and deploy density (boot time 
and footprint)

> >      2.3. Setup exception traps.
> >
> > 3. Support single CPU.
> >
> > 4. Support multiple threads.
> >      4.1. Implement GIC interrupt controller drivers. If we doesn't specify
> the gic version in QEMU's parameter,
> >           default GIC will be detected by kvm_arm_vgic_probe. Most ARM hosts
> are using GICv2, GICv3 and GICv4,
> >           and QEMU will provide GICv2 and GICv3 emulators. For best
> compatibility, we have to implement gicv2
> >           and gicv3 drivers without MSI/MSI-X support. This means we don't
> need to implement gicv2m, gicv3-its
> >           for Unikraft in this time.
> >      4.2. Implment ARMv8 virtual timer driver.
> >
> > 5. Setup a 1:1 mapping pagetable for Physical memory and Virtual memory.
> >      5.1. Configure MMU
> >      5.2. Create page tables with 1GB or 2MB block
> >
> > 6. Implement PSCI interface to support machine shutdown.
> 
> FWIW, system_off only exist from PSCI 0.2 and onwards.
> 

It seem the psci-0.2 is the default PSCI version of mach-virt with KVM.

> >
> > 7. Network, block and etc IO devices?
> >     Should we have to port virtual device driver like virtio-net, pv-net
> from KVM and Xen?
> 
> There are no emulation provided on Xen, so you would need PV drivers to
> get access to the network/block.

Yes, I have the same opinion with you 😊


> 
> Cheers,
> 
> --
> Julien Grall
_______________________________________________
Minios-devel mailing list
Minios-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/minios-devel