Re: [Minios-devel] [UNIKRAFT PATCHv4 21/43] plat/kvm: Add Arm64 basic entry code

[Wei Chen <Wei.Chen@xxxxxxx>]

Hi Julien,

> -----Original Message-----
> From: Julien Grall <julien.grall@xxxxxxx>
> Sent: 2018年7月16日 18:51
> To: Wei Chen <Wei.Chen@xxxxxxx>; minios-devel@xxxxxxxxxxxxxxxxxxxx;
> simon.kuenzer@xxxxxxxxx
> Cc: Kaly Xin <Kaly.Xin@xxxxxxx>; nd <nd@xxxxxxx>
> Subject: Re: [Minios-devel] [UNIKRAFT PATCHv4 21/43] plat/kvm: Add Arm64 basic
> entry code
> 
> Hi Wei,
> 
> On 16/07/18 07:42, Wei Chen wrote:
> >> -----Original Message-----
> >> From: Julien Grall <julien.grall@xxxxxxx>
> >> Sent: 2018年7月13日 18:42
> >> To: Wei Chen <Wei.Chen@xxxxxxx>; minios-devel@xxxxxxxxxxxxxxxxxxxx;
> >> simon.kuenzer@xxxxxxxxx
> >> Cc: Kaly Xin <Kaly.Xin@xxxxxxx>; nd <nd@xxxxxxx>
> >> Subject: Re: [Minios-devel] [UNIKRAFT PATCHv4 21/43] plat/kvm: Add Arm64
> basic
> >> entry code
> >>
> >>
> >>
> >> On 13/07/18 11:11, Wei Chen wrote:
> >>> Hi Julien,
> >>
> >> Hi Wei,
> >>
> >>>> -----Original Message-----
> >>>> From: Julien Grall <julien.grall@xxxxxxx>
> >>>> Sent: 2018年7月12日 18:06
> >>>> To: Wei Chen <Wei.Chen@xxxxxxx>; minios-devel@xxxxxxxxxxxxxxxxxxxx;
> >>>> simon.kuenzer@xxxxxxxxx
> >>>> Cc: Kaly Xin <Kaly.Xin@xxxxxxx>; nd <nd@xxxxxxx>
> >>>> Subject: Re: [Minios-devel] [UNIKRAFT PATCHv4 21/43] plat/kvm: Add Arm64
> >> basic
> >>>> entry code
> >>>>
> >>>>
> >>>>
> >>>> On 12/07/18 10:43, Wei Chen wrote:
> >>>>> Hi Julien,
> >>>>
> >>>> Hi Wei,
> >>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Julien Grall <julien.grall@xxxxxxx>
> >>>>>> Sent: 2018年7月11日 23:52
> >>>>>> To: Wei Chen <Wei.Chen@xxxxxxx>; minios-devel@xxxxxxxxxxxxxxxxxxxx;
> >>>>>> simon.kuenzer@xxxxxxxxx
> >>>>>> Cc: Kaly Xin <Kaly.Xin@xxxxxxx>; nd <nd@xxxxxxx>
> >>>>>> Subject: Re: [Minios-devel] [UNIKRAFT PATCHv4 21/43] plat/kvm: Add
> Arm64
> >>>> basic
> >>>>>> entry code
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 11/07/18 10:50, Wei Chen wrote:
> >>>>>>> Hi Julien,
> >>>>>>
> >>>>>> Hi Wei,
> >>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Julien Grall <julien.grall@xxxxxxx>
> >>>>>>>> Sent: 2018年7月8日 6:24
> >>>>>>>> To: Wei Chen <Wei.Chen@xxxxxxx>; minios-devel@xxxxxxxxxxxxxxxxxxxx;
> >>>>>>>> simon.kuenzer@xxxxxxxxx
> >>>>>>>> Cc: Kaly Xin <Kaly.Xin@xxxxxxx>; nd <nd@xxxxxxx>
> >>>>>>>> Subject: Re: [Minios-devel] [UNIKRAFT PATCHv4 21/43] plat/kvm: Add
> >> Arm64
> >>>>>> basic
> >>>>>>>> entry code
> >>>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> On 07/06/2018 10:03 AM, Wei Chen wrote:
> >>>>>>>>> QEMU/KVM can boot an Arm64 elf image without multiboot. In this 
> >>>>>>>>> case,
> >>>>>>>>> we can plage _libkvmplat_entry to entry64.S directly as the vCPU
> >>>>>>>>> reset entry. In this basic entry code, we just initialize the boot
> >>>>>>>>> stack and prepare jumping to _libkvmplat_start.
> >>>>>>>> Can you clarify why you are using the ELF format and not Image? My
> main
> >>>>>>>> concern is the former does not seem to have a clear description of
> the
> >>>>>>>> state of the VM at boot.
> >>>>>>>>
> >>>>>>>
> >>>>>>> It's little hard for me to answer your question. This is why I reply
> >> this
> >>>>>>> Comment at the last. Actually, when I was selecting the elf image I
> >> didn’t
> >>>>>>> think so much. And most Unikernel projects that I have involved (ukvm,
> >>>> mini-
> >>>>>> os)
> >>>>>>> are using the elf image, both for arm and x86.
> >>>>>>
> >>>>>> Mini-OS ARM is using the zImage format, not ELF. For UKVM, IIRC, you
> >>>>>
> >>>>> Yes, arm32 is zImage, but x86_64 is using OUTPUT_FORMAT("elf64-x86-64").
> >>>>> And about the ukvm, Yes, I wrote it by myself, because the ukvm requires
> >> elf
> >>>>> format, it only support elf loader. We want to make our Unikernel
> >>>> application
> >>>>> like a normal elf application that can run on virtual machine directly.
> We
> >>>> don't
> >>>>> want to be compatible with the Linux image boot protocol, it's too
> complex
> >>>> for us.
> >>>>> And we don't have some many information need to pass.
> >>>>
> >>>> Why is it too complex? The Image format is basically a couple fields to
> >>>> slap at the top of your binary and a few guidelines for the tools how to
> >>>> deal with the VM state. Most of the boot code will be the same.
> >>>>
> >>>> Also, most likely if you are going to support kvmtools, you would need
> >>>> to use the Image format. For Xen, support for ELF will require a lot of
> >>>> reworks and to be honest it is not worth the effort (I tried it before).
> >>>>
> >>>
> >>> I have read the Xen code and Shijie's Arm64 mini-OS patches. Shijie was
> >>> Using elf format as image.From his code, I didn't see lots of reworks
> >>> That I need to do If I will enable elf format for Xen?
> >> Arm64 Mini-OS is not using ELF. Xen tools does not have ELF support for
> >> Arm, and you can look at the patch [1] to see the image been created
> >> using objcopy.
> >>
> >
> > Ok, you're right, xen tools doesn't support. but he was using the ELF format
> in
> > linking stage. Here is Shijie's branch on GITHUB:
> > https://github.com/zyzii/mini-os/blob/arm64-v3/arch/arm/arm64/minios-
> arm64.lds.S
> 
> Because EFL format is the standard for building C application. Both Xen
> and Linux does exactly the same.
> 
> >
> >>
> >>>
> >>> Kvmtools doesn't support Arm elf loader. But it supports other
> architectures.
> >>> And as your said, if we using the kernel image format? How about ukvm,
> >>> it requires elf format? Relative to kvmtools, I am more like to support
> ukvm.
> >>> Because in a way, qemu and kvmtools are overlapping. Actually, I don't
> mind
> >> to
> >>> provide different image format for different platform.
> >>
> >> May I ask why did you decide to require using ELF for UKVM? What was the
> >> rationale? Is that just because x86 was doing it?
> >
> > Yes, x86 is using the elf format too. About rationale? I haven't asked
> > Mato.
> 
> This is usually the standard to boot OS for x86. Xen and KVM does
> support the same for x86.
> 
> However, on Arm64, I haven't seen many people requesting ELF so far. I
> actually dropped the support for Arm in Xen because it was rotten.
> 
> >
> >>
> >> But then why using that for QEMU? You need to compile your ELF assuming
> >
> > Can I ask you why QEMU supports elf format image? If QEMU support it, why I
> > can't use it? While I was implementing the Arm64 enablement, elf format is
> > the simplest way for me to verify my code. I just need to place my code at
> > the entry point, then QEMU would help to load it to CPU reset entry.
> 
> Mostly likely because it was support on x86 and was easy to add support
> for Arm.
> 
> But as I said earlier, I am not against using ELF. However, there need
> to be some documentation telling you how to boot. At the moment, it is
> close to zero. So can you write down the expectation?

I think I can write down expectation in another improvement patch series,
but not this series. While I was writing this basic entry code, I didn't
think so much. I just wanted Unikraft to be enabled on Arm64 ASAP, even
this code contains some bugs. Let's open another separate thread and patch
series to improve it.

> >
> >> VA == PA, which is rather fragile. You also yet haven't told me what is
> >> the expect ABI (e.g state of the machine...) for ELF.
> >>
> >
> > I don't think is fragile. My application has only one memory address space,
> > VA == PA is mandatory. Most of the application developers will not touch the
> > link script. My expect state of the machine is MMU off on reset.
> >
> >>>
> >>>>>
> >>>>>> wrote it yourself. So I guess you based your understanding of the state
> >>>>>> of the VM from somewhere?
> >>>>>>
> >>>>>
> >>>>> Each SoC, include QEMU virtual machine, they will give most system
> >> registers
> >>>>> a default value. I am using cortex-a53 VCPU, the QEMU gives it a reset
> >> value
> >>>>> to cpu->reset_sctlr = 0x00c50838; MMU, I/D cache are disabled.
> >>>>> But yes, you're right, I ignore the MMU has been disabled already, and
> >>>> disable
> >>>>> It again in entry code.
> >>>>
> >>>> This looks very fragile to rely on for a specific processor. What if we
> >>>> decide to use another Cortex-A*? Or even another Arm CPU?
> >>>>
> >>>> We need a clear definition of the VM state. For instance, you say the
> >>>> cache is disabled. Has the kernel image/DT been cleaned to PoC by the
> >>>> tools? You will also have to be careful when writing the page-table as
> >>>> "Cache disabled" does not rely mean "disabled". It is more a by-pass of
> >>>> the cache.
> >>>>
> >>>
> >>> I can read them from sctlr if I want. I am curious about how does other
> >>> Bootloader handle such case? Don't think my code is an entry code for
> >>> OS. Let's treat it as a bootloader.
> >>
> >> I don't understand what you mean. Whether your code is a bootloader or
> >> an OS, you have to know exactly what is the state of your platform when
> >> the processor jumps to your entry code.
> >>
> >
> > I just know my MMU and Caches are disabled while I was writing the code.
> > I admit my MMU disable code in entry is unnecessary.
> >
> >> Your Image has been loaded by QEMU in the memory. If you modify the
> >> page-table with cache disabled and not clean the DT. Then you may end up
> >> using stall information when re-enabling the cache.
> >
> > What does the DT mean here? Device tree? I haven't re-enable the cache, I
> > just enable it only once.
> 
> It looks like I mangle two sentence. I meant cleaning the cache. See
> more below.
> 
> >
> >>
> >> If you disable the MMU, then this will still have an impact on your
> >> cache as well. This is because KVM has alias to that region you will
> >> modify and therefore you may have stall data in your cache when turning
> >> on the MMU.
> >>
> >
> > Even I think this scenario could not happen in current code, I still
> > Want to know how to avoid such problem before I re-enable the MMU?
> Can you explain why do you think it can't happen? KVM has all the memory
> mapped with cacheable attributes. So you have a cacheable alias on your
> guest memory.
> 

Thanks for your explanation. About the memory attributes, I remember last
year, I asked a question about what will happen will guest and host have
different memory attributes in Linux-eng. I remember the answer is to follow
the more restricted attributes. KVM mapped the memory as cacheable, but
guest disable the cache through system registers. So I think guest memory
is non-cacheable.
Can I understand the "cacheable alias" as "data existed in cache of guest
memory after KVM mapping, but before VM start?"

> However, when you will write page-table, you will write with Device
> nGnRnE attributes (because MMU is disabled). So the cache will be
> bypassed. The cache may still contain stall data that you will hit when
> enabling MMU and cache.
> 
> To prevent such issue, you need to clean the cache potentially before
> and after updating the page-table area. I also mention before because it
> looks like the page-table will not be part of the kernel (the region is
> not populated), and therefore the cache state is unknown.
> 

Hmm, I understand your concern know. But I have a question. Should we do
such operations on bare metal? I had written code for several bare
metals before. All of these bare metals' MMU are off when reset, I hadn't
do above operations. If so, can I understand that QEMU-KVM and bare metals
have different behaviors for same code? In this case, how can we run
unmodified code on QEMU-KVM?

As my understanding, while QEMU is creating a CPU, if it disable the cache
in SCTLR, it would clean the cache, if not it would be a bug of QEMU.

How about Xen handle such case?


> I would have recommended you to look at the Linux code, but that's GPL
> code and you write BSD code :). Maybe someone in your team can look at
> Linux and summarize the problem and way to fix it to you?
> 

The "someone" can't be anyone but me ; (

> >
> >> Overall, I think you should treat Unikraft as an OS as usually the
> >> format will give you more guaranty than the platform itself. This will
> >> also allow you to re-use that code accross multiple environment rather
> >> than tailoring for QEMU.
> >
> > Yes, I agree with you. This is why Simon and I decide to create a
> plat/common
> > Folder. We want to share most code as we can among different platforms.
> > But in current stage, QEMU is our first target, we're focusing on enable
> > and verify features on QEMU platform, and then port to other platforms.
> >
> > About the image format, I understand your concern. But I prefer to use
> > another patch series to change it to improve its compatibility. If we
> > support ukvm later, it will have a separate elf format image.
> 
> That should be fine, as long as you write down the assumptions and get
> the boot code correct.
> 
> Cheers,
> 
> --
> Julien Grall
_______________________________________________
Minios-devel mailing list
Minios-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/minios-devel