Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Mon, 27 Nov 2023, Chuck Zmudzinski wrote:
> On 11/27/2023 10:22 AM, Chuck Zmudzinski wrote:
> > On 11/27/2023 7:45 AM, Mario Marietto wrote:
> >> @Chuck Zmudzinski <mailto:brchuckz@xxxxxxxxxxxx> : Stay tuned. They want 
> >> to help us. The xen developers are great. Very good support for us. I'm 
> >> sure that you can give a good contribution to understand what's our 
> >> problem and how to implement a fix with the help of all those good guys.
> >> 
> >> On Mon, Nov 27, 2023 at 11:56 AM Roger Pau Monné <roger.pau@xxxxxxxxxx 
> >> <mailto:roger.pau@xxxxxxxxxx>> wrote:
> >> 
> >>     On Mon, Nov 27, 2023 at 10:28:13AM +0000, Henry Wang wrote:
> >>     > +(xen-devel and Arm maintainers, including Julien)
> >>     >
> >>     > > On Nov 27, 2023, at 18:03, Mario Marietto <marietto2008@xxxxxxxxx 
> >> <mailto:marietto2008@xxxxxxxxx>>
> >>     > > wrote:
> >>     > >
> >>     > > Hello.  We have just virtualized Debian 12 on our arm (32 bit)
> >>     > > Chromebook model xe303c12 . As host / dom0 we have chosen Devuan
> >>     > > 5,and for guest / domU,Debian 12. It works great. But our goal is
> >>     > > different. We want to virtualize FreeBSD as domU. Can we have a
> >>     > > working Xen PV network driver for a FreeBSD arm guest ?. I found
> >>     > > that Julien Grall has ported the Xen drivers to FreeBSD on arm. I
> >>     > > would like to know if Julien's work was accepted upstream by
> >>     > > FreeBSD, in which case FreeBSD as a Xen guest on arm should work
> >>     > > if we enable the Xen PV drivers in the FreeBSD on arm kernel. If
> >>     > > Julien's work was not accepted upstream by FreeBSD, we will have
> >>     > > to find his patches and apply them ourselves to the FreeBSD on arm
> >>     > > kernel.
> >> 
> >>     I've added Elliot on Cc as he is working on upstreaming the patches to
> >>     FreeBSD.  He will be able to provide a better update than myself.
> >> 
> >>     Regards, Roger.
> > 
> > I have been collaborating with Mario, and I can explain what we have done 
> > so far :
> > 
> > We are using Julien's patch set against an old development version of 
> > FreeBSD 11
> > from 2014-12-03 :
> > 
> > https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm-v2.2
> > 
> > We successfully built the XENVIRT kernel and FreeBSD world, and created the
> > FreeBSD rootfs according to Julien's instructions here :
> > 
> > https://lists.freebsd.org/pipermail/freebsd-xen/2014-November/002202.html
> > 
> > There were some adjustments to the instructions :
> > 
> > To build the kernel, we used :
> > 
> > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel
> > 
> > instead of
> > 
> > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel
> > 
> > The FreeBSD 'kernel' file is in ELF format and did not work, and we spent
> > some time trying to convert it to the zImage format without realizing the
> > build of the FreeBSD kernel creates the 'kernel.bin' file in the zImage 
> > format.
> > So when booting with the 'kernel.bin' file instead, it actually boots :
> > 
> > user@devuan-bunsen ~ % sudo xl create freebsd.cfg
> > Parsing config from freebsd.cfg
> > user@devuan-bunsen ~ % sudo xl li
> > Name                                        ID   Mem VCPUs  State   Time(s)
> > Domain-0                                     0   768     2     r-----    
> > 1439.4
> > freebsd                                      1  1152     1     r-----       
> > 3.0
> > user@devuan-bunsen ~ %
> > 
> > However, the guest is still not working correctly :
> > 
> > 1. Attaching the console with the -c option at creation or with
> >    'xl console freebsd' results in no output to the console.
> > 
> > 2. The timestamp on the virtual disk image file shows that the filesystem
> >    was at best mounted read-only, if it was mounted at all by the guest
> >    FreeBSD kernel.
> > 
> > 3. The 'xl shutdown freebsd' command does not work, it just hangs. To stop
> >    the guest, you need to do 'xl destroy freebsd'.
> > 
> > However, I think we can get the console to work and the rootfs to mount 
> > because I
> > just realized I forgot to do the steps from Julien's instructions of 
> > editing the
> > /etc/fstab and /etc/ttys files in the FreeBSD rootfs :
> > 
> > $ echo "/dev/xbd0       /       ufs     rw      1       1" > /mnt/etc/fstab
> > $ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on 
> > secure")
> > 
> > I will add those and see if the console and disk are working.
> 
> Unfortunately, adding xc0 to /etc/ttys and /dev/xbd0 as the root device in
> /etc/fstab did not make the console or disk work. Still no output on the
> xen console from the guest kernel, and the timestamp on the rootfs image
> file did not change so it did not mount read-write.
> 
> We could use some advice for troubleshooting this. Now, we are blind because
> we are not getting any xen console output But I am pleased we were able to
> demonstrate that Julien's old patch set for FreeBSD 11 allows us to boot
> FreeBSD 11 on a modern version of Xen on arm - we are using the Debian
> stable Xen 4.17 packages.

You can use the DEBUG hypercalls to check how far we got into the
booting process:
https://wiki.xenproject.org/wiki/Xen_ARM_DEBUG_hypercalls

For instance add the following to FreeBSD code:

asm volatile("hvc 0xfffd");