Re: AMD EPYC virtual network performances

[Elliott Mitchell <ehem+xen@xxxxxxx>]

On Tue, Jul 09, 2024 at 08:36:18AM +0000, Andrei Semenov wrote:
> 
> After some investigations we notices a huge performances drop (perfs divided
> by
> factor of 5) starting from 5.10.88 Linux kernel version on the AMD EPYC
> platforms. The patch introduced in this kernel version that allows to
> pinpoint
> the buggy behavior is :
> 
>  “xen/netfront: harden netfront against event channel storms”
> d31b3379179d64724d3bbfa87bd4ada94e3237de
> 
> The patch basically binds the network frontend to the `xen_lateeoi_chip`
> irq_chip (insead of `xen_dynamic_chip`) which allows to its clients to
> inform
> the chip if spurious interrupts are detected and so the delay in interrupt
> treatment is introduced by the chip.

I worry I'm being knave here.

For the heck of it, I took a glance at b27d47950e48.  If my understanding
is correct, b27d47950e48 is making a very basic (and wrong) assumption
about timing/latency.

In particular any time either side receive an event, it will handle
X # of incoming payloads and Y # of acknowledged outgoing payloads.  As
such if X + Y > 1, then up to X + Y - 1 spurious events may be detected.
The issue is there is no synchronization between the event channel and
the work queues.

In particular the network back end could legitimately generate:

work0   signal0 work1   signal1 work2   signal2 work3   signal3

Whereas the network front end may handle this as:

event0  work0   work1   work2   work3   event1  event2  event3

Where b27d47950e48 would interpret events 1-3 as spurious, even though
they're perfectly legitimate.  The same phenomenon could occur in both
directions and also with the Xen block devices.

Ultimately how is the network portion of XSA-391 any different from any
other network DoS?  If an interrupt is generated for every single packet
of a series of runt frames, there will be heavy processor use for little
network traffic.

AMD systems may fair worse than Intel systems due to differing cache
coherence behavior/latency.  Perhaps AMD's NUMA implementation adds
some latency.  (huh, suddenly the RAID1 issue comes to mind)


Hopefully I'm not making knave speculation here.  Might this be the
simplest of issues, just it was missed due to being too obvious?


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@xxxxxxx  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445