Re: next meeting: Monday June 16th 10:00 - 12:00 CEST

[Hannes Mehnert <hannes@xxxxxxxxxxx>]

Dear everybody,

last week we had a nice discussion. The next one is in one week (June 
30th 10:00 - 12:00 CEST at https://meet.jit.si/MirageOS -- see our 
shared pad at https://pad.data.coop/To6IOSeNSOK9kFVlgo7XWw?both# for 
notes and agenda (add your talking points there) :))

Below are the notes from last week.

Best,

Hannes

## Meeting June 16th 10:00 - 12:00 CEST
- Participants: Pierre, Reynir, Hannes, Sam

### IO-page without Cstruct, and Cstruct.t without Bigarray
- Pierre: have a branch with no bigarray in mirage-tcpip (see 
https://github.com/hannesm/mirage-tcpip/tree/no-bigarray)
```
cstruct.6.2.0                                git 
git+https://github.com/hannesm/ocaml-cstruct.git#no-bigarray
cstruct-lwt.6.2.0                            git 
git+https://github.com/hannesm/ocaml-cstruct.git#no-bigarray
io-page.3.0.0                                git 
git+file:///home/user/mirage/io-page#no-cstruct
mirage-net-xen.2.1.5                         git 
git+file:///home/user/mirage/mirage-net-xen#io-page-ba
mirage-xen.9.0.0                             git 
git+file:///home/user/mirage/mirage-xen#new-iopage
shared-memory-ring.3.2.1                     git 
git+file:///home/user/mirage/shared-memory-ring#new-iopage
shared-memory-ring-lwt.3.2.1                 git 
git+file:///home/user/mirage/shared-memory-ring#new-iopage
tcpip.9.0.1                   (uninstalled)  git 
git+https://github.com/hannesm/mirage-tcpip.git#no-bigarray
vchan.6.0.2                                  git 
git+file:///home/user/mirage/ocaml-vchan#update-iopage-api
vchan-xen.6.0.2                              git 
git+file:///home/user/mirage/ocaml-vchan#update-iopage-api
xen-gnt.4.0.2                 (uninstalled)  git 
git+file:///home/user/mirage/ocaml-gnt#new-iopage
```
```
[2025-06-16 10:00:52]             |      ___|
[2025-06-16 10:00:52]   __|  _ \  |  _ \ __ \
[2025-06-16 10:00:52] \__ \ (   | | (   |  ) |
[2025-06-16 10:00:52] ____/\___/ _|\___/____/
[2025-06-16 10:00:52] Solo5: Bindings version v0.9.1
[2025-06-16 10:00:52] Solo5: Memory map: 32 MB addressable:
[2025-06-16 10:00:52] Solo5:   reserved @ (0x0 - 0xfffff)
[2025-06-16 10:00:52] Solo5:       text @ (0x100000 - 0x498fff)
[2025-06-16 10:00:52] Solo5:     rodata @ (0x499000 - 0x55afff)
[2025-06-16 10:00:52] Solo5:       data @ (0x55b000 - 0x8e4fff)
[2025-06-16 10:00:52] Solo5:       heap >= 0x8e5000 < stack < 0x2000000
[2025-06-16 10:00:52] 2025-06-16T08:00:52-00:00: [INFO] [net-xen 
frontend] connect 0
[2025-06-16 10:00:52] 2025-06-16T08:00:52-00:00: [INFO] [qubes.db] 
connecting to server...
[2025-06-16 10:00:52] 2025-06-16T08:00:52-00:00: [INFO] [qubes.db] connected
[2025-06-16 10:00:52] 2025-06-16T08:00:52-00:00: [INFO] [net-xen 
frontend] create: id=0 domid=1
[2025-06-16 10:00:52] 2025-06-16T08:00:52-00:00: [INFO] [net-xen 
frontend]  sg:true gso_tcpv4:true rx_copy:true rx_flip:false 
smart_poll:false
[2025-06-16 10:00:52] 2025-06-16T08:00:52-00:00: [INFO] [net-xen 
frontend] MAC: 00:16:3e:5e:6c:00
[2025-06-16 10:00:53] 2025-06-16T08:00:53-00:00: [INFO] [ethernet] 
Connected Ethernet interface 00:16:3e:5e:6c:00
[2025-06-16 10:00:53] 2025-06-16T08:00:53-00:00: [INFO] [ipv6] IP6: Starting
[2025-06-16 10:00:53] 2025-06-16T08:00:53-00:00: [INFO] [ARP] Sending 
gratuitous ARP for 10.137.0.27 (00:16:3e:5e:6c:00)
[2025-06-16 10:00:53] Fatal error: exception 
Invalid_argument("String.blit / Bytes.blit_string")
[2025-06-16 10:00:53] Raised at Stdlib.invalid_arg in file "stdlib.ml", 
line 30, characters 20-45
[2025-06-16 10:00:53] Called from 
Ethernet__Ethernet_packet.Marshal.unsafe_fill in file 
"duniverse/ethernet/src/ethernet_packet.ml", line 63, characters 4-67
[2025-06-16 10:00:53] Called from Stdlib__Result.map in file 
"result.ml", line 25, characters 32-37
[2025-06-16 10:00:53] Called from Ethernet.Make.write.fill in file 
"duniverse/ethernet/src/ethernet.ml", line 107, characters 14-60
[2025-06-16 10:00:53] Called from 
Frontend.Make.write_already_locked.(fun) in file 
"duniverse/mirage-net-xen/lib/frontend.ml", line 304, characters 18-60
[2025-06-16 10:00:53] Called from Lwt.Sequential_composition.try_bind in 
file "duniverse/lwt/src/core/lwt.ml", line 2139, characters 10-14
[2025-06-16 10:00:53] Re-raised at Lwt.Miscellaneous.poll in file 
"duniverse/lwt/src/core/lwt.ml", line 3123, characters 20-29
[2025-06-16 10:00:53] Called from Xen_os__Main.run.aux in file 
"duniverse/mirage-xen/lib/main.ml", line 37, characters 10-20
[2025-06-16 10:00:53] Called from Dune__exe__Main.run in file 
"mirage/main.ml", line 4, characters 12-29
[2025-06-16 10:00:53] Called from Dune__exe__Main in file 
"mirage/main.ml", line 436, characters 2-7
[2025-06-16 10:00:53] Solo5: solo5_exit(2) called
```
- The issue is likely in mirage-net-xen passing a (too) small buffer to 
Ethernet, since the source location is a blit of the destination mac 
address (at offset 0 into the buffer)
- Pierre: need to also investigate performance, since with the Cstruct 
using bytes does more copying
- Hannes: I'm really happy this is moving forward.
- Hannes: the ethernet implementation calls `Netif.write`, which calls 
the (mirage-net-xen) Frontend.write with the `~size` parameter, which 
then calls `write_no_retry`, which calls `write_already_locked` 
(`Cstruct.sub`) -- and I guess we should printf the `~size` parameter 
for further debugging
- Hannes: in (mirage-net-xen) `write_already_locked` we call `fillf 
(Cstruct.sub cs_shared_block 0 size)`
- Hannes: in (ethernet):  `let size = eth_hdr_size + size`
- Hannes: I'd recommend to printf the `size`
- Pierre: there's some too small buffer passed somewhere
- Hannes: I'm really happy to see this moving forward. The blocker in 
Mirleft was clearly the hanging of qubesdb/xen vchan. I'd love to think 
further about performance, and it seems the roadblock is lifted now :)
### Unikraft
- Sam: The ocaml-unikraft package family has been released to the 
opam-repository
- Virgile told that opam-monorepo is complaining with how the 
ocaml-unikraft packages are setup
- Sam: we have patches for the mirage tool, also for net, block, time, 
etc. - we try to fix the small issues that are left, and it will be 
ready soon!
- Hannes: did you do any further benchmarking?
- Sam: not yet, there was a small bug: there are some extra bytes in 
each packet - unclear where they are coming from. we can safely just 
ignore them, they are always null bytes.
- Sam: we plan to run again some benchmarks this week, and hope is that 
they'll be reasonably good
- Pierre: in the end, it will be a potential replacement for solo5?
- Sam: exactly, it is an alternative backend. for unikraft, you don't 
need solo5 tenders - you can run on qemu and firecracker. they have 
experimental ports for other backends they are working on.
- Pierre: would it be possible to use with xen and qubes, or not?
- Sam: unikraft has a backend for xen, but we didn't add it to the other 
layers - since I don't have a path to test it
- Sam: I didn't write the code to generate the xen backend (in 
ocaml-unikraft)
- Pierre: would be interesting, esp. if it delivers better performance
- Hannes: which network stack do you use?
- Sam: we initially used the unikraft one (which uses lwip), but now we 
use the OCaml one
- Hannes: any news on multiple CPUs in unikraft?
- Sam: not yet. it is on their roadmap. it is not in any released 
version. in some parts of the code I needed to debug, it is clear that 
it is monocore
- Hannes: what is unikraft currently working on? do they have users?
- Sam: they created a company to smoothen the deployment experience 
(unikraft cloud)
- Hannes: I'm really happy you're pushing this work forward, so we can 
piggyback on the unikraft cloud, and at the same time we can evaluate 
the performance issues from their website (where solo5 is really slow), 
and figure out whether the performance is lost in solo5 or OCaml.
### OCaml compiler and cross-compiling
- Sam: The OCaml 5.5 will include the patches needed for 
cross-compiling, so we don't need any further patches \o/ (apart from 
setting the number of domains to 1)
- Hannes: For OCaml 5.4, do we have a patchset?
- Sam: there's only a single PR left (to enable freestanding target), 
will open a PR on ocaml-solo5 and ocaml-unikraft when 5.4 is released
- Hannes: from the OCaml release schedule, 5.4 is expected this summer 
(August)
### OxCaml
- JaneStreet published their OxCaml project (including stack allocation, 
etc.) at https://oxcaml.org
- MirageOS could use Oxcaml - it includes as well flambda2
- It would be interesting / nice to test the various compilers:
  - OCaml, OCaml with flambda1, OxCaml (with plain opam packages), 
OxCaml with annotated packages (to use e.g. stack allocation)
- Sam: it is unclear how to use OxCaml with MirageOS, they seem to be 
based on the 5.2.0 compiler - and there would be various cross-compiling 
PRs needed to use OxCaml inside of MirageOS
- Sam: unclear whether it is worth to backpatch OxCaml, or wait until 
they move to OCaml 5.3.0 or even 5.4.0. Maybe we'll find someone who has 
an idea of their roadmap.
- Sam: it shouldn't be too hard to get something working, but it would 
be more work to upstream clean patches as merged.