Re: Issue: Networking performance in Xen VM on Arm64

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Tue, 11 Oct 2022, Leo Yan wrote:
> > > The second question is how to mitigate the long latency when send data
> > > from DomU.  A possible solution is the Xen network forend driver copies
> > > skb into mediate (bounce) buffer, just like what does in Xen net
> > > backend driver with gnttab_batch_copy(), in this way the forend driver
> > > doesn't need to wait for backend driver response and directly return
> > > back.
> > 
> > About this, I am not super familiar with drivers/net/xen-netfront.c but
> > I take you are referring to xennet_tx_buf_gc? Is that the function that
> > is causing xennet_start_xmit to wait?
> 
> No.  We can take the whole flow in xen-netfront.c as:
> 
>   xennet_start_xmit()
>              ----------> notify Xen Dom0 to process skb
>              <---------  Dom0 copies skb and notify back to DomU
>   xennet_tx_buf_gc()
>   softirq/NET_TX : __kfree_skb()

Let me premise again that I am not an expert in PV netfront/netback.
However, I think the above is only true if DomU and Dom0 are running on
the same physical CPU. If you use sched=null as I suggested above,
you'll get domU and dom0 running at the same time on different physical
CPUs and the workflow doesn't work as described.

It should be:

CPU1: xennet_start_xmit()             ||  CPU2: doing something else
CPU1: notify Xen Dom0 to process skb  ||  CPU2: receive notification
CPU1: return from xennet_start_xmit() ||  CPU2: Dom0 copies skb
CPU1: do something else               ||  CPU2: notify back to DomU
CPU1: receive irq, xennet_tx_buf_gc() ||  CPU2: do something else


> > I didn't think that waiting for the backend is actually required. I
> > mean, in theory xennet_start_xmit could return without calling
> > xennet_tx_buf_gc, it is just an optimization. But I am not sure about
> > this.
> 
> The function xennet_start_xmit() will not wait and directly return
> back, but if we review the whole flow we can see the skb is freed until
> the softirq NET_TX.

Is it an issue that the skb is not freed until later? Is that affecting
the latency results? It shouldn't, right? What matters is when dom0 is
getting those packets on the physical network interface and that happens
before the skb is freed. I am just trying to figure out if we are
focusing on the right problem.


> In this whole flow, it needs DomU and Dom0 to work
> together (includes two context switches) to process skb.

There are not necessarily 2 context switches as things should run in
parallel.


> Here I mean the optimization is to allow Dom0 and DomU to work in
> parallel.  It could be something like blow, the key point is DomU
> doesn't need to wait for Dom0's notification.

I think it is already the case that domU doesn't need to wait for dom0's
notification? It is true that domU is waiting for dom0's notification to
free the skb but that shouldn't affect latency?


>        DomU                     |             Dom0
>   ------------------------------+-------------------------------
>   xennet_start_xmit()           |
>   copy skb in grant page        |
>   notify Xen Dom0               |
>                                 |  fetch skb from grant page
>   xennet_tx_buf_gc()            |  deliver skb to bridge
>     kfree_skb()                 |