Re: Issue: Networking performance in Xen VM on Arm64

[Julien Grall <julien@xxxxxxx>]

Hi Stefano,

On 18/10/2022 20:54, Stefano Stabellini wrote:
> On Tue, 18 Oct 2022, Leo Yan wrote:
> > On Mon, Oct 17, 2022 at 04:50:05PM -0700, Stefano Stabellini wrote:
> >
> > [...]
> >
> > > > Which means it takes 543us to let Dom0 to receive the notification.
> > > > You could see DomU runs in CPU3 and Dom0 runs on CPU13, there should
> > > > not have contention for CPU resources.  Seems to me, it's likely Xen
> > > > hypervisor takes long time to deliver the interrupt, note, it's not
> > > > take so long time for every skb transferring, sometimes the time for
> > > > response a notification is short (about ~10us).
> > > Good find. I think this is worth investigating further. Do you have
> > > vwfi=native in your Xen command line as well?
> > Yes, I have added "sched=null" and "vwfi=native" into Xen options:
> >
> > options=noreboot dom0_mem=4096M bootscrub=0 iommu=on loglvl=error 
> > guest_loglvl=error sched=null vwfi=native
> >
> > > After that, I would add printk also in Xen with the timestamp. The event
> > > channel notification code path is the following:
> > >
> > > # domU side
> > > xen/arch/arm/vgic-v2.c:vgic_v2_to_sgi
> > > xen/arch/arm/vgic.c:vgic_to_sgi
> > > xen/arch/arm/vgic.c:vgic_inject_irq
> > > xen/arch/arm/vgic.c:vcpu_kick
> > > xen/arch/arm/gic-v2.c:gicv2_send_SGI
> > >
> > > # dom0 side
> > > xen/arch/arm/gic.c:do_sgi
> > > xen/arch/arm/traps.c:leave_hypervisor_to_guest
> > >
> > > It would be good to understand why sometimes it takes ~10us and some
> > > other times it takes ~540us
> > Thanks a lot for detailed info.
> >
> > Just note, in my platform DomU enables GICv3 driver rather than GICv2.
> > This would be a bit different in the Xen code.  But it should be easy
> > for me to map to vgic-v3 files.
> >
> > I have a question for how to trace Xen system.  Outputting chars to UART
> > is time costy (usually it's millisecond level), it is not friendly to
> > use console for debugging performance issue.  I searched a bit, either
> > "xl dmesg" or xentrace can be used for capturing trace logs, one thing
> > I am not certain is if we can save Xen logs only into log buffer and
> > doesn't output to UART, so that afterwards we can use "xl dmesg" to
> > capture the logs.  Could anyone confirm for this is correct usage with
> > "xl dmesg" or I should use xentrace for this case?
> I am not super familiar about this. But if you take away the console
> entirely from Xen and give it to dom0 only (no dtuart command line
> option to Xen), you should get the behavior you described:
> - no console output because Xen has no console
> - xl dmesg should still work
You are right, xl dmesg will still work. The console is built using a 
ring so if you are using "xl dmesg" you will only get the latest log in 
the ring.
xenconsoled has an option to save the hypervisor log in 
/var/log/xen/hypervisor.log (default path) if you request to dump them. 
This is an option at startup and can be set in the init.d/systemd 
configuration for Xen (see XENCONSOLED_TRACE).
That said, xentrace is better suited if you are looking for low overhead 
tracing. The console is not directly involved in the process and you can 
already benefits from the trace point sprinkled in Xen code (such as the 
scheduling).
Cheers,

--
Julien Grall