Re: arm: DomU Networking enable leads to Dom0 kernel oops

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

Hi Andrei,

Reply at the bottom.


On Fri, 24 Apr 2020, Andrei Cherechesu wrote:
> Hello,
> 
>  
> 
> Recently I’ve been trying to enable the networking in a conventional DomU 
> (not dom0less).
> 
> The approach I used was the one described here: 
> https://wiki.xen.org/wiki/Xen_Networking#Bridging.
> 
>  
> 
> But when I use xl to create DomU, I get a Kernel OOPS in Dom0. The setup is 
> still responsive after this
> 
> and DomU boots successfully. However, if I try to enable `eth0` in DomU 
> (using ip link dev set eth0 up),
> 
> I get a Kernel Panic in Dom0.
> 
>  
> 
> Here are the complete steps:
> 
>  1. Boot Dom0 and configure bridge:
>      +  brctl addbr xenbr0
>      +  brctl addif xenbr0 eth0
>      +  ip link set dev xenbr0 up
>      +  ip link set dev eth0 up
>      +  dhclient xenbr0
> 
>  
> 
>  2. ifconfig xenbr0 (ping 8.8.8.8 works):
> 
> xenbr0    Link encap:Ethernet  HWaddr B2:62:37:C9:BB:D7
> 
>           inet addr:192.168.0.185  Bcast:192.168.0.255  Mask:255.255.255.0
> 
>           inet6 addr: fe80::b062:37ff:fec9:bbd7/64 Scope:Link
> 
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
> 
>           RX packets:703 errors:0 dropped:0 overruns:0 frame:0
> 
>           TX packets:27 errors:0 dropped:0 overruns:0 carrier:0
> 
>           collisions:0 txqueuelen:1000
> 
>           RX bytes:88213 (86.1 KiB)  TX bytes:3088 (3.0 KiB)
> 
>  
> 
>  3. ifconfig eth0:
> 
> eth0      Link encap:Ethernet  HWaddr B2:62:37:C9:BB:D7 
> 
>           inet6 addr: fe80::b062:37ff:fec9:bbd7/64 Scope:Link
> 
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
> 
>           RX packets:1535 errors:0 dropped:2 overruns:0 frame:0
> 
>           TX packets:43 errors:0 dropped:0 overruns:0 carrier:0
> 
>           collisions:0 txqueuelen:1000
> 
>           RX bytes:450476 (439.9 KiB)  TX bytes:4248 (4.1 KiB)
> 
>           Interrupt:59 Base address:0xc000
> 
>  
> 
>  4. Add bridge to DomU config file:
> 
> vif = [ 'bridge=xenbr0' ]
> 
>  
> 
>  5. root@s32g274aevb-Dom0:~# xl create /etc/xen/domU1.cfg
> 
> After this I get the Kernel OOPS, because of a failed memory
> 
> access
> 
> [  413.367873] Unable to handle kernel paging request at virtual address 
> 000006c000020070
 

[...]
 
> Do you have any idea why this occurs? Have I misconfigured anything?

The configuration looks correct. I have not seen this issue before, and
it goes without saying that it should not happen. PV network (vif) is
used very often so I was surprised to see an issue like this.

Then I saw Julien's reply where he noticed that you are using LinuxRT in
Dom0. I tried that once before in the last few months and I had issues
with it (issues with LinuxRT running as dom0 kernel; as domU works
fine). I had to go back to using vanilla Linux in dom0 and LinuxRT in
DomU only.  Thank you for reminding me of the problem :-)

I went back and checked my configuration, using LinuxRT in dom0 again and
trying to start a VM with PV network like you did. In my case, dom0
doesn't crash but prints the error appended below and domU hangs at
boot.  Dom0 and DomU are both 4.19.0-rt1-00002-gf6cb7e02d291. Then I
tried using upstream 5.6.4-rt3, and interestingly I got the same error
again.

But I get the same problem with or without vif=[""] in the config file,
and also this seems to be an interrupt related issue, while the one you
are seeing seems to be a memory mapping issue, so they are very
different.

I wonder why we are seeing two completely different errors. It might be
because of differences in our Linux config, or it could be due to
hardware differences. I think it makes sense for you to try a more
recent version of LinuxRT, ideally 5.6.4-rt3 like I did. Once you do
that, could you please share both your findings and your kernel .config
file? (I attached mine for 5.6.4-rt3 as a reference.)

We should be able to get to the point where we are both seeing the same
problem :-)



[   86.900974] ------------[ cut here ]------------
[   86.905134] Interrupt for port 6, but apparently not enabled; per-user 
(____ptrval____)
[   86.913228] WARNING: CPU: 0 PID: 2437 at drivers/xen/evtchn.c:167 
evtchn_interrupt+0xfc/0x108
[   86.913231] Modules linked in:
[   86.913240] CPU: 0 PID: 2437 Comm: irq/55-evtchn:x Not tainted 
4.19.0-rt1-00002-gf6cb7e02d291 #57
[   86.913242] Hardware name: ZynqMP ZCU102 Rev1.0 (DT)
[   86.913246] pstate: 80000005 (Nzcv daif -PAN -UAO)
[   86.913251] pc : evtchn_interrupt+0xfc/0x108
[   86.913256] lr : evtchn_interrupt+0xfc/0x108
[   86.913258] sp : ffffff8009f43d70
[   86.913260] x29: ffffff8009f43d70 x28: ffffffc03d045cb0 
[   86.913266] x27: ffffff80080f0000 x26: ffffff8008f1d000 
[   86.913272] x25: ffffffc03d045c9c x24: ffffff80080f0668 
[   86.913277] x23: ffffffc03d045c00 x22: 0000000000000001 
[   86.913283] x21: 0000000000000037 x20: ffffffc03c980800 
[   86.913288] x19: ffffffc03db64c00 x18: ffffff8008f1d688 
[   86.913293] x17: 0000000000000001 x16: 0000000000000007 
[   86.913299] x15: ffffff8088fc1167 x14: 0000000000000006 
[   86.913304] x13: ffffff8008fc1175 x12: 73752d726570203b 
[   86.913309] x11: 0000000000000000 x10: 0000000005f5e0ff 
[   86.913314] x9 : ffffff8009f43a50 x8 : 2820726573752d72 
[   86.913320] x7 : 0000000000000000 x6 : ffffff8008f1d688 
[   86.913325] x5 : 0000000000000003 x4 : 00000040570b3000 
[   86.913330] x3 : 0000000000000001 x2 : 0000000000000001 
[   86.913335] x1 : d7954973361fc800 x0 : 0000000000000000 
[   86.913341] Call trace:
[   86.913346]  evtchn_interrupt+0xfc/0x108
[   86.913352]  irq_forced_thread_fn+0x2c/0x90
[   86.913355]  irq_thread+0x104/0x1e0
[   86.913361]  kthread+0xf8/0x128
[   86.913366]  ret_from_fork+0x10/0x1c
[   86.913369] ---[ end trace 0000000000000002 ]---

Attachment: .config
Description: Text document