Re: [Minios-devel] Some considerations of ARM Unikraft supports

[Wei Chen <Wei.Chen@xxxxxxx>]

Hi Simon,

> -----Original Message-----
> From: Simon Kuenzer [mailto:simon.kuenzer@xxxxxxxxx]
> Sent: 2018年2月6日 0:21
> To: Wei Chen <Wei.Chen@xxxxxxx>; Julien Grall <julien.grall@xxxxxxxxxx>
> Cc: Felipe Huici <Felipe.Huici@xxxxxxxxx>; Kaly Xin <Kaly.Xin@xxxxxxx>; Shijie
> Huang <Shijie.Huang@xxxxxxx>; Florian Schmidt <Florian.Schmidt@xxxxxxxxx>;
> Costin Lupu <costin.lup@xxxxxxxxx>; nd <nd@xxxxxxx>; minios-
> devel@xxxxxxxxxxxxx
> Subject: Re: [Minios-devel] Some considerations of ARM Unikraft supports
> 
> Hi Wei, hi Julien,
> 
> thanks a lot for discussing this already, I put my comments inline.
> 
> On 05.02.2018 08:22, Wei Chen wrote:
> > Hi Julien,
> >
> > Thanks for your comments!
> > Replies inline.
> >
> >> -----Original Message-----
> >> From: Julien Grall [mailto:julien.grall@xxxxxxxxxx]
> >> Sent: 2018年2月2日 18:43
> >> To: Wei Chen <Wei.Chen@xxxxxxx>; Simon Kuenzer <simon.kuenzer@xxxxxxxxx>
> >> Cc: Felipe Huici <Felipe.Huici@xxxxxxxxx>; Kaly Xin <Kaly.Xin@xxxxxxx>;
> Shijie
> >> Huang <Shijie.Huang@xxxxxxx>; Florian Schmidt <Florian.Schmidt@xxxxxxxxx>;
> >> Costin Lupu <costin.lup@xxxxxxxxx>; nd <nd@xxxxxxx>; minios-
> >> devel@xxxxxxxxxxxxx
> >> Subject: Re: [Minios-devel] Some considerations of ARM Unikraft supports
> >>
> >> Hi,
> >>
> >> On 02/02/18 09:10, Wei Chen wrote:
> >>> This week I am trying to boot Unikraft on ARM64/KVM platform. In this
> >> progress I have
> >>> got some considerations and written a simple proposal:
> >>>
> >>> My first target is to enable Unikraft on ARM64+Kvm, so this proposal would
> >> focus on ARM64+Kvm.
> >>> But the goal of ARM support is to enable Unikraft on ARM32/ARM64 based
> >> hypervisors (ARM32/64 Kvm,
> >>> ARM64 Xen and etc). So we have to consider to keep current multi-arch
> >> framework and reuse common
> >>> code like virtual drivers for ARM32/ARM64.
> >>>
> >>> 1. Modify the folders for multi-architectures
> >>>       1.1. Add arm64 folder to unikraft/arch:
> >>>                unikraft----arch----arm
> >>>                              |-----x86_64
> >>>                              |-----arm64 <-- New
> >>>
> >>>            Above folders contains architecture specified Makefile, Config,
> >> Compiler flags and some
> >>>            code. In most cases, these files are exclusive. So we'd better
> >> keep each arcitecture in
> >>>            a standalone floder. This also can avoid doing to much changes
> to
> >> Unikraft Makefile.
> >>>
> >>>            If we add arm64 to unikraft/arch/arm, we have to do more ARCH
> >> comparasion in Makefile:
> >>>                unikraft----arch----arm----arm32
> >>>                          |      |-----arm64 <-- New
> >>>                        |
> >>>                          |-----x86_64
> >>>            Before:$(UK_BASE)/arch/$(ARCH)/Makefile.uk.
> >>>            After:$(UK_BASE)/arch/arm/$(ARCH)/Makefile.uk
> >>>            This change is complex, so we'd better to add arm64 folder to
> >> unikraft/arch.
> >>
> >> Except the assembly code, most of the C code should be very similar
> >> between ARM64 and ARM32. So it might make more sense to have a directory
> >> arch/arm with sub-folder arm32 and arm64.
> >>
> >
> > This is one option I had considered. But this will add a new variable
> (VENDOR) to
> > make scripts. e.g. :$(UK_BASE)/arch/$(VENDOR)/$(ARCH)/Makefile.uk
> > And currently, only architecture dependent code will be placed in $(ARCH)
> folder.
> > For example, in arm folder, there are some files for arm32 math library.
> These
> > files can only be used for arm32.
> 
> What is this vendor variable about? Is it something that applies to a
> specific silicon? Is it required to add subfolders for it?
> 

Yes, it applies to a specific silicon. But "VENDOR" is not very accurate here.
I had considered it again, because x86 is not a "VENDOR", and not all x86 chips
Belong to intel, Maybe use "FAMILY" is better.

If we really have some common C code for ARM32/64, I agree to add subfolders
for it.

unikraft----arch----arm----arm32  ARM family arm32 and arm64 architectures
             |       |-----arm64
             |
             |------x86----i386
                     |-----x86_64 X86 family i386 and x86_64 architectures

> >
> > If some C codes are very similar between arm32 and arm64, I think this code
> would
> > be very similar between arm and x86 too. We can place these codes in
> Unikraft/lib.
> >
> > Above 2 options would affect the common framework, so I still want to get
> some
> > Comments from Simon.
> 
> I welcome this discussion because one of the exercises of Unikraft's 0.2
> releases is to figure out how to do the right split.
> I am okay with changing the structure of the arch folder substructure if
> we can foresee already that it will make sense. In such a case, I would
> also like to adopt the same principle to the x86 architecture folder.
> 
> The idea of architecture libraries is that they contain code which is
> only special to the CPU but the same to all of the target platforms
> (xen, kvm, linux). We were originally expecting that this is mostly
> assembly code but we might be wrong with our original assumption. So, if
> you foresee any common C code for 32 and 64bit ARM that would be
> duplicated otherwise, we should use a single arm folder instead.
> 

Sorry, about " use a single arm folder instead". Does it mean we don't add
Any subfolders to arm or x86 folder? Like following?

unikraft----arch----arm
             |       
             |------x86
                     
> >
> >>>
> >>>       1.2. Add arm64 to unikraft/include/uk/arch
> >>>
> >>>       1.3. Add arm64 kvm platform code to unikraft/plat/kvm/arm, and use
> >> Makefile to select
> >>>            objects for correct architecutre:
> >>>
> >>>            ifeq ($(ARCH_X86_64),y)
> >>>               LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/x86/entry64.S
> >>>               LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/x86/cpu_x86_64.c
> >>>            else ifeq ($(ARCH_ARM_64),y)
> >>>               LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/entry64.S
> >>>               LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/cpu_arm64.c
> >>>            else ifeq ($(ARCH_ARM_64),y)
> >>>               LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/entry.S
> >>>               LIBKVMPLAT_SRCS-y += $(LIBKVMPLAT_BASE)/arm/cpu_arm.c
> >>>            endif
> >>>
> >>>       1.4. Add a "drivers" folder to unikraft/
> >>>            This because we may have some virtual device drivers can be
> shared
> >> among platforms.
> >>>            For example, we can reuse virtual uart, timer and gic drivers
> from
> >> arm32/arm64 Kvm/xen.
> 
> Is it okay for you to wait with the driver folder a bit? I am currently
> working on PCI for x86 KVM and I figured that Unikraft need an mechanism
> to select drivers for devices (and maybe buses) individually for each
> platform. But drivers are still something that depend on the platform.
> For instance Xen could reuse the same PCI drivers with pcifront, linux
> with VFIO, but a third platform might not support PCI at all.
> 
> Because of this, I am currently considering to introduce an folder in
> plat: e.g., plat/common/drivers/pci/virtio-net. What do you guys think?
> 

That's quite good, I will wait it : )

> >>>
> >>> 2. Bootloader
> >>>       2.1. Because of the BIOS, x86 is using multiboot to load kernel on
> >> Linux-KVM QEMU. But on ARM platforms,
> >>>            we can skip the EFI and boot from the Virtual Machine's RAM
> base
> >> address. So we can place _libkvmplat_entry
> >>>            to the CPU's reset entry by link script. On ARM64 platform, the
> >> default virtual machine CPU model is cortex A15.
> >>
> >> Cortex A15 does not support 64-bit. So how come it is the default
> >> virtual machine CPU model for ARM64?
> >>
> >
> >  From the code, if we don't specify any cpumodel, the mach-virt's default
> > cpumodel will be set to "cortex-a15". But you'are right, if we use cortex-15
> > by default, we can run any 64-bit image. Here is my mistake. We have to set
> > correct cpumodel (cortex-a53/a57 or host) in command line to make 64-bit
> image
> > work. But the mach-virt is still using the a15memmap and a15irqmap.
> >
> >
> >> But likely, you want to expose the same MIDR as the underlying CPU. So
> >> if an errata has to be implemented in Unikraft, it will be able to know it.
> >>
> >
> > Exposing the underlying CPU's MIDR to guest is depending on the hypervisors.
> > For Unikraft itself, it doesn't know whether this MIDR is the same as the
> underlying
> > CPU or not. And actually, no matter what cpumodel the hypervisor is
> emulating, the
> > code is running on the physical CPU directly. We don't emulate the CPU
> instructions.
> > If we run Unikraft on a corext-a53 host CPU, we can compile this image with
> gcc flags
> > like fix-a53-error.
> >
> >>>
> >>>            plat/kvm/arm/link64.ld:
> >>>            ENTRY(_libkvmplat_entry)
> >>>            SECTIONS {
> >>>                . = 0x40000000;
> >>>
> >>>                /* Code */
> >>>                _stext = .;
> >>>
> >>>                .text :
> >>>                {
> >>>                    *(.text)
> >>>                    *(.text.*)
> >>>                }
> >>>
> >>>                _etext = .;
> >>>                ...
> >>>            }
> >>>
> >>>       2.2. Use the fixed physical addresses of PL011 uart, timer and GIC.
> So
> >> we can skip the device tree parse.
> >>
> >> What does promise you the PL011, timer, GIC will always be at the same
> >> address?
> >
> > My original idea was that we selected a fixed machine (mach-virt) for
> Unikraft to run.
> > In this case, the memory map is fixed.
> >
> >> Or do you expect the user to hack unikraft build system to set
> >> the address?
> >>
> >
> > For my opinion, Yes. Why should we need to parse the device tree to increase
> our boot
> > time and footprint?
> >
> 
> Sorry for my stupid question: Would this hardcode the guest device
> configuration that you would need to set with KVM? I mean, how are
> network devices (or other) are handover to the guest? If yes, I am
> concerned that Unikraft is getting difficult to use on ARM. I would
> rather prefer to provide a configuration option where users could
> disable that the image scans the device tree and expects devices at
> hardcoded places.

While I was writing this proposal, I hadn't consider so many devices. I just
considered some platform devices like interrupt controller, timer and UART.
At that moment, I prefer to hardcode. But now I think parse the device tree
is better. Because the virtual net/block devices are dynamic configuration
for a VM.

> 
> >> At least from Xen PoV, the memory layout is not part of the ABI and a
> >> guest should rely on the DT for getting the correct addresses.
> >>
> >
> > I understand your concern. It's not a part of the ABI. So the addresses can
> be changed
> > for different boards.
> >
> > I think we must do a tradeoff between flexibility and deploy density (boot
> time and footprint)
> >
> 
> If this makes sense for you: I prefer having the most flexible as
> default and provide configuration options with Config.uk to switch them
> off individually. I think Unikraft should handover such tradeoff
> question to Unikernel builders.
> 

That would be good.

> >>>       2.3. Setup exception traps.
> >>>
> >>> 3. Support single CPU.
> 
> This is fine for the first version. The other platforms also just
> support a single CPU for now.
> 
> >>>
> >>> 4. Support multiple threads.
> >>>       4.1. Implement GIC interrupt controller drivers. If we doesn't
> specify
> >> the gic version in QEMU's parameter,
> >>>            default GIC will be detected by kvm_arm_vgic_probe. Most ARM
> hosts
> >> are using GICv2, GICv3 and GICv4,
> >>>            and QEMU will provide GICv2 and GICv3 emulators. For best
> >> compatibility, we have to implement gicv2
> >>>            and gicv3 drivers without MSI/MSI-X support. This means we
> don't
> >> need to implement gicv2m, gicv3-its
> >>>            for Unikraft in this time.
> >>>       4.2. Implment ARMv8 virtual timer driver.
> >>>
> 
> Please contact Costin what is required from the Unikraft's scheduler
> API. I CC'ed him.
> 

Thanks, I will contact Costin when I start to implement this driver.

> >>> 5. Setup a 1:1 mapping pagetable for Physical memory and Virtual memory.
> >>>       5.1. Configure MMU
> >>>       5.2. Create page tables with 1GB or 2MB block
> >>>
> 
> Good.
> 
> >>> 6. Implement PSCI interface to support machine shutdown.
> >>
> >> FWIW, system_off only exist from PSCI 0.2 and onwards.
> >>
> >
> > It seem the psci-0.2 is the default PSCI version of mach-virt with KVM.
> >
> >>>
> >>> 7. Network, block and etc IO devices?
> >>>      Should we have to port virtual device driver like virtio-net, pv-net
> >> from KVM and Xen?
> 
> After we agreed how Unikraft should include drivers we can start with
> porting them. Is KVM on ARM using virtio-net, too? Is there a virtual
> PCI bus attached?

Yes, KVM on ARM is using virtio-net too. The virtio-net is connect to a
virtio-mmio bus. But there is a ECAM PCI host controller emulator too.

> 
> >>
> >> There are no emulation provided on Xen, so you would need PV drivers to
> >> get access to the network/block.
> 
> This is fine ;-).
> 
> >
> > Yes, I have the same opinion with you 😊
> >
> >
> >>
> >> Cheers,
> >>
> >> --
> >> Julien Grall
> 
> Thanks,
> 
> Simon
_______________________________________________
Minios-devel mailing list
Minios-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/minios-devel