Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Mario Marietto <marietto2008@xxxxxxxxx>]

Hello.

I know two FreeBSD developers that can commit your patches,if they think that they are good. I could drop an email to them. I've hardly helped one of them with the testing when we have implemented the passthrough of a recent nvidia gpu on Linux. The second one is experienced with arm and virtualization. Plus I know the main developer of bhyve. I'm happy to ask them to review your work. 

On Tue, Nov 28, 2023 at 12:04 AM Elliott Mitchell <ehem+freebsd@xxxxxxx> wrote:

On Mon, Nov 27, 2023 at 10:57:42AM -0500, Chuck Zmudzinski wrote:
> On 11/27/2023 10:22 AM, Chuck Zmudzinski wrote:
> >
> > I have been collaborating with Mario, and I can explain what we have done so far :
> >
> > We are using Julien's patch set against an old development version of FreeBSD 11
> > from 2014-12-03 :
> >
> > https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm-v2.2
> >
> > We successfully built the XENVIRT kernel and FreeBSD world, and created the
> > FreeBSD rootfs according to Julien's instructions here :
> >
> > https://lists.freebsd.org/pipermail/freebsd-xen/2014-November/002202.html
> >
> > There were some adjustments to the instructions :
> >
> > To build the kernel, we used :
> >
> > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel
> >
> > instead of
> >
> > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel
> >
> > The FreeBSD 'kernel' file is in ELF format and did not work, and we spent
> > some time trying to convert it to the zImage format without realizing the
> > build of the FreeBSD kernel creates the 'kernel.bin' file in the zImage format.
> > So when booting with the 'kernel.bin' file instead, it actually boots :
> >
> > user@devuan-bunsen ~ % sudo xl create freebsd.cfg
> > Parsing config from freebsd.cfg
> > user@devuan-bunsen ~ % sudo xl li
> > Name                                        ID   Mem VCPUs  State   Time(s)
> > Domain-0                                     0   768     2     r-----    1439.4
> > freebsd                                      1  1152     1     r-----       3.0
> > user@devuan-bunsen ~ %
> >
> > However, the guest is still not working correctly :
> >
> > 1. Attaching the console with the -c option at creation or with
> >    'xl console freebsd' results in no output to the console.
> >
> > 2. The timestamp on the virtual disk image file shows that the filesystem
> >    was at best mounted read-only, if it was mounted at all by the guest
> >    FreeBSD kernel.
> >
> > 3. The 'xl shutdown freebsd' command does not work, it just hangs. To stop
> >    the guest, you need to do 'xl destroy freebsd'.
> >
> > However, I think we can get the console to work and the rootfs to mount because I
> > just realized I forgot to do the steps from Julien's instructions of editing the
> > /etc/fstab and /etc/ttys files in the FreeBSD rootfs :
> >
> > $ echo "/dev/xbd0       /       ufs     rw      1       1" > /mnt/etc/fstab
> > $ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on secure")
> >
> > I will add those and see if the console and disk are working.
>
> Unfortunately, adding xc0 to /etc/ttys and /dev/xbd0 as the root device in
> /etc/fstab did not make the console or disk work. Still no output on the
> xen console from the guest kernel, and the timestamp on the rootfs image
> file did not change so it did not mount read-write.
>
> We could use some advice for troubleshooting this. Now, we are blind because
> we are not getting any xen console output But I am pleased we were able to
> demonstrate that Julien's old patch set for FreeBSD 11 allows us to boot
> FreeBSD 11 on a modern version of Xen on arm - we are using the Debian
> stable Xen 4.17 packages.

The only method for booting I ever tried was Tianocore/EDK2.  As brought
up previously, that needs an update for Xen 4.17 though:
https://lists.xenproject.org/archives/html/xen-devel/2023-10/msg00278.html
Essentially they had two uses of the shared info page, one was unmapped
and the other never got unmapped.  That is good enough for FreeBSD's
bootloader to function.

FreeBSD's handling of patches is unfortunately very friendly to people
who have permission to commit, or know someone responsible for a
particular area.  If you're trying to get patches for something which
has shared responsibility and you're unfunded you're SoL.  As such my
work has basically stalled.  Everything works, but without anyone with
an interest it is simply sitting here mouldering.  Combined with the
annoyance curse it is simply sitting.

Julien Grall's patches are very much PoC.  As such I've done a lot of
updating.  Take a look at branch "submit":
https://gitlab.com/ehem/freebsd-src.git

Problem is FreeBSD's interrupt situation is troublesome.  Rather than 1
interrupt framework, there are 4.  Each has different built-in
assumptions.  "INTRNG" was designed for ARM and deliberately threw away
the x86 assumptions, but then added other assumptions.

The tip of the "submit" branch has a mix of different potential
approaches to working with "INTRNG".  Several of these might be rejected,
but hopefully one might be accepted.  Really I need someone in the
FreeBSD organization with the authority to override Michal Meloun's
proselytizing about INTRNG being perfect.

The other problem is no one in the FreeBSD organization wants to deal
with it's interrupt disaster.  By the time anyone is allowed to commit,
they've been thoroughly burned by the situation and are unwilling to
touch the interrupt system.  Thus leading to the antipattern of it being
just barely maintained.

I've got it working, just I'm stuck.


BTW Roger Pau Monné, now that Xen 4.18 is out, take a look at the
"royger" branch?


--
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@xxxxxxx  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

--

Mario.