Need help debugging kernel oops ( BUG: unable to handle page fault for address ? )

["J. Roeleveld" <joost@xxxxxxxxxxxx>]

Hi All,

 

I have had a few oopses in the past week already and am trying to find out what the likely cause is and, more importantly, how to resolve this.

 

I have tried google, but not found anything useful yet. Several results showing issues during boot (not the case as it runs succesfully for nearly a week) or related to 2.6 kernel versions.

 

I have attached the dmesg-output after the first time I noticed the "oops" below this email. (Actually, several in a row)

Last nights was unable to get as the server was frozen by the time I got to it. Which means I am unable to confirm fully if the pattern was the same. The last message I could still read was nearly identical to the ones I saw the first time.

 

I noticed the following section which looks interesting:

===

[317321.524229] BUG: unable to handle page fault for address: ffff888510ebd0e0

[317321.524307] #PF: supervisor write access in kernel mode

[317321.524368] #PF: error_code(0x0003) - permissions violation

===

But I have no idea if this is a cause or a result of the earlier trace messages in the output.

 

I found a new BIOS and Firmware version available for the mainboard, which I am planning on applying this week.

 

The kernel is "tainted" because of the use of ZFS. No other out-of-tree modules are installed.

 

My distro: Gentoo

Kernel version: 5.4.38

ZFS version: 0.8.3

XEN version: 4.12.2

 

If more info is needed to analyse this, please let me know.

 

Additionally, if anyone has/knows good resources (online preferred, but hardcopy will be fine as well) I can use to analyse/understand these kernel messages I would definitely appreciate it.

 

Many thanks in advance,

 

Joost Roeleveld

 

DMESG:

 

[317321.523586] ------------[ cut here ]------------

[317321.523600] WARNING: CPU: 1 PID: 25465 at arch/x86/xen/multicalls.c:102 xen_mc_flush+0x194/0x1c0

[317321.523601] ------------[ cut here ]------------

[317321.523603] Modules linked in:

[317321.523614] WARNING: CPU: 3 PID: 2162 at arch/x86/xen/multicalls.c:102 xen_mc_flush+0x194/0x1c0

[317321.523615] iscsi_tcp libiscsi_tcp libiscsi

[317321.523618] Modules linked in:

[317321.523619] scsi_transport_iscsi nfsd

[317321.523622] iscsi_tcp

[317321.523623] auth_rpcgss nfs_acl lockd

[317321.523626] libiscsi_tcp

[317321.523627] grace sunrpc br_netfilter xt_physdev xen_acpi_processor

[317321.523631] libiscsi

[317321.523632] xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev

[317321.523636] scsi_transport_iscsi

[317321.523637] xen_evtchn xenfs xen_privcmd bridge 8021q

[317321.523640] nfsd

[317321.523642] garp mrp stp llc

[317321.523644] auth_rpcgss

[317321.523646] bonding intel_rapl_msr iTCO_wdt

[317321.523648] nfs_acl

[317321.523650] iTCO_vendor_support intel_rapl_common sb_edac intel_powerclamp

[317321.523653] lockd

[317321.523654] crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel intel_rapl_perf

[317321.523658] grace

[317321.523659] pcspkr i2c_i801 ast

[317321.523662] sunrpc

[317321.523663] drm_vram_helper ttm i2c_algo_bit drm_kms_helper

[317321.523666] br_netfilter

[317321.523667] drm lpc_ich mei_me

[317321.523670] xt_physdev

[317321.523671] mei ipmi_ssif ixgbe mdio

[317321.523674] xen_acpi_processor

[317321.523675] ptp ioatdma pps_core dca ipmi_si

[317321.523679] xen_pciback

[317321.523680] ipmi_devintf ipmi_msghandler acpi_power_meter binfmt_misc

[317321.523683] xen_netback

[317321.523684] tun

[317321.523686] xen_blkback

[317321.523687] zfs(PO) zunicode(PO)

[317321.523689] xen_gntalloc

[317321.523690] zavl(PO) icp(PO)

[317321.523692] xen_gntdev

[317321.523694] zcommon(PO) znvpair(PO) spl(O)

[317321.523696] xen_evtchn

[317321.523698] zlua(PO)

[317321.523700] xenfs xen_privcmd bridge 8021q

[317321.523707] CPU: 1 PID: 25465 Comm: zfs_stats_cache Tainted: P O 5.4.38-gentoo-host #1

[317321.523708] garp

[317321.523710] Hardware name: Supermicro Super Server/X10DRi-T4+, BIOS 3.1 06/08/2018

[317321.523711] mrp

[317321.523715] RIP: e030:xen_mc_flush+0x194/0x1c0

[317321.523716] stp llc bonding

[317321.523721] Code: 05 00 10 00 81 e8 ec 13 be 00 48 89 c1 48 89 45 18 48 c1 e9 3f 48 89 ce e9 03 ff ff ff 48 c7 45 18 ea ff ff ff be 01 00 00 00 <0f> 0b 8b 55 00 48 c7 c7 a0 8a fb 81 31 db 65 8b 0d e7 0a ff 7e e8

[317321.523722] intel_rapl_msr iTCO_wdt iTCO_vendor_support

[317321.523726] RSP: e02b:ffffc9000a00bbe8 EFLAGS: 00010002

[317321.523727] intel_rapl_common sb_edac

[317321.523730] intel_powerclamp crct10dif_pclmul crc32_pclmul

[317321.523736] RAX: ffff888686655858 RBX: 0000777f80000000 RCX: ffff888686655858

[317321.523737] crc32c_intel ghash_clmulni_intel

[317321.523741] RDX: 0000000000000001 RSI: 000000000000000d RDI: ffff888686655310

[317321.523742] intel_rapl_perf pcspkr i2c_i801 ast

[317321.523747] RBP: ffff888686655300 R08: ffff888686655810 R09: 0000000000000001

[317321.523749] drm_vram_helper ttm

[317321.523753] R10: 0000000000007ff0 R11: 00000000000c8ea9 R12: 0000000000000000

[317321.523753] i2c_algo_bit drm_kms_helper

[317321.523758] R13: ffff8885924ef000 R14: 0000000000000201 R15: 0000000000000015

[317321.523759] drm lpc_ich mei_me mei ipmi_ssif ixgbe mdio ptp ioatdma pps_core dca ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter binfmt_misc tun zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlua(PO)

[317321.523784] FS: 00007f5ddb094b80(0000) GS:ffff888686640000(0000) knlGS:0000000000000000

[317321.523788] CS: 10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033

[317321.523793] CPU: 3 PID: 2162 Comm: kworker/3:2 Tainted: P O 5.4.38-gentoo-host #1

[317321.523795] Hardware name: Supermicro Super Server/X10DRi-T4+, BIOS 3.1 06/08/2018

[317321.523798] CR2: 0000559802a28270 CR3: 0000000566956000 CR4: 0000000000040660

[317321.523815] Workqueue: events netstamp_clear

[317321.523818] Call Trace:

[317321.523823] RIP: e030:xen_mc_flush+0x194/0x1c0

[317321.523827] Code: 05 00 10 00 81 e8 ec 13 be 00 48 89 c1 48 89 45 18 48 c1 e9 3f 48 89 ce e9 03 ff ff ff 48 c7 45 18 ea ff ff ff be 01 00 00 00 <0f> 0b 8b 55 00 48 c7 c7 a0 8a fb 81 31 db 65 8b 0d e7 0a ff 7e e8

[317321.523836] __xen_pgd_unpin+0xf8/0x210

[317321.523838] RSP: e02b:ffffc90004097d08 EFLAGS: 00010002

[317321.523844] xen_exit_mmap+0xdb/0xf0

[317321.523847] RAX: ffffffffffffffea RBX: 000000059ca3e000 RCX: 0000000000000001

[317321.523859] exit_mmap+0x66/0x1a0

[317321.523863] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8886866d5b10

[317321.523869] ? __schedule+0x213/0x5c0

[317321.523872] RBP: ffff8886866d5300 R08: 0000000000000000 R09: ffff88867ec01c38

[317321.523878] mmput+0x52/0x110

[317321.523880] R10: 0000000000007ff0 R11: ffffffff8224c7a8 R12: 0000000000000000

[317321.523884] flush_old_exec+0x261/0x680

[317321.523887] R13: 0000000000000020 R14: ffff88867a04b740 R15: 0000000000000001

[317321.523893] load_elf_binary+0x31f/0x16a0

[317321.523897] ? get_user_pages_remote+0x142/0x230

[317321.523900] ? _cond_resched+0x15/0x30

[317321.523906] ? selinux_inode_permission+0x107/0x1d0

[317321.523912] ? load_misc_binary+0x4c/0x410 [binfmt_misc]

[317321.523915] search_binary_handler+0x84/0x1b0

[317321.523918] FS: 00007f5ddb094b80(0000) GS:ffff8886866c0000(0000) knlGS:0000000000000000

[317321.523921] CS: 10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033

[317321.523925] CR2: 0000559801ac9f64 CR3: 000000059ca3e000 CR4: 0000000000040660

[317321.523927] __do_execve_file.isra.0+0x502/0x870

[317321.523930] __x64_sys_execve+0x35/0x40

[317321.523936] do_syscall_64+0x5b/0x1a0

[317321.523941] entry_SYSCALL_64_after_hwframe+0x44/0xa9

[317321.523943] Call Trace:

[317321.523946] RIP: 0033:0x7f5ddb19af87

[317321.523956] xen_write_cr3+0xf0/0x160

[317321.523966] load_new_mm_cr3+0x39/0xc0

[317321.523966] Code: Bad RIP value.

[317321.523968] RSP: 002b:00007ffccf1b4ea8 EFLAGS: 00000202 ORIG_RAX: 000000000000003b

[317321.523971] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f5ddb19af87

[317321.523976] switch_mm_irqs_off+0x199/0x4f0

[317321.523978] RDX: 0000559802a26ff0 RSI: 0000559802a276d0 RDI: 0000559802a26f50

[317321.523983] ? netif_rx_internal+0x1f/0x100

[317321.523985] RBP: 0000559802a26f50 R08: 0000559802a276d0 R09: 0000559802a28270

[317321.523987] R10: 00007f5ddb3429b0 R11: 0000000000000202 R12: 00000000ffffffff

[317321.523988] R13: 0000559802a276d0 R14: 0000559802a26ff0 R15: 0000559802a26dd0

[317321.524000] __text_poke+0x20e/0x4a0

[317321.524004] ---[ end trace 1ddd52a17333fad8 ]---

[317321.524009] text_poke_bp_batch+0xab/0x150

[317321.524016] arch_jump_label_transform_apply+0x2e/0x50

[317321.524018] 13 of 21 multicall(s) failed: cpu 1

[317321.524020] call 3: op=14 arg=[ffff88851054d000] result=-16

[317321.524022] call 4: op=14 arg=[ffff88859697c000] result=-16

[317321.524029] static_key_disable_cpuslocked+0x59/0x90

[317321.524031] call 6: op=14 arg=[ffff88850b2c2000] result=-16

[317321.524034] static_key_disable+0x16/0x20

[317321.524036] call 8: op=14 arg=[ffff888510ebd000] result=-16

[317321.524042] process_one_work+0x1e5/0x3a0

[317321.524044] call 10: op=14 arg=[ffff8884caee2000] result=-16

[317321.524045] call 11: op=14 arg=[ffff88857d673000] result=-16

[317321.524048] worker_thread+0x4d/0x3d0

[317321.524050] call 12: op=14 arg=[ffff8885440d5000] result=-16

[317321.524055] kthread+0xfb/0x130

[317321.524057] call 14: op=14 arg=[ffff8884d6dc7000] result=-16

[317321.524059] call 16: op=14 arg=[ffff8884cedd7000] result=-16

[317321.524062] ? process_one_work+0x3a0/0x3a0

[317321.524066] ? kthread_park+0x90/0x90

[317321.524068] call 17: op=14 arg=[ffff888506dd7000] result=-16

[317321.524070] call 18: op=14 arg=[ffff888673189000] result=-16

[317321.524076] ret_from_fork+0x35/0x40

[317321.524077] call 20: op=14 arg=[ffff8886375cc000] result=-16

[317321.524079] call 21: op=14 arg=[ffff88859ca3e000] result=-16

[317321.524083] ---[ end trace 1ddd52a17333fad9 ]---

[317321.524090] 1 of 1 multicall(s) failed: cpu 3

[317321.524093] call 1: op=26 arg=[ffff8886866d5b10] result=-22

[317321.524229] BUG: unable to handle page fault for address: ffff888510ebd0e0

[317321.524307] #PF: supervisor write access in kernel mode

[317321.524368] #PF: error_code(0x0003) - permissions violation

[317321.524411] ------------[ cut here ]------------

[317321.524423] PGD 220c067 P4D 220c067 PUD 5fdb45067 PMD 5fdabd067 PTE 8010000510ebd065

[317321.524510] WARNING: CPU: 3 PID: 2162 at arch/x86/xen/multicalls.c:102 xen_mc_flush+0x194/0x1c0

[317321.524573] Oops: 0003 [#1] SMP NOPTI

[317321.524692] Modules linked in: iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd auth_rpcgss nfs_acl lockd grace sunrpc br_netfilter xt_physdev xen_acpi_processor xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn xenfs xen_privcmd bridge 8021q garp mrp stp llc bonding intel_rapl_msr iTCO_wdt iTCO_vendor_support intel_rapl_common sb_edac intel_powerclamp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel intel_rapl_perf pcspkr i2c_i801 ast drm_vram_helper ttm i2c_algo_bit drm_kms_helper drm lpc_ich

[317321.524730] CPU: 1 PID: 25465 Comm: zfs_stats_cache Tainted: P W O 5.4.38-gentoo-host #1

[317321.524730] mei_me mei ipmi_ssif ixgbe mdio ptp ioatdma pps_core dca ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter binfmt_misc tun zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlua(PO)

[317321.525152] Hardware name: Supermicro Super Server/X10DRi-T4+, BIOS 3.1 06/08/2018

[317321.525288] CPU: 3 PID: 2162 Comm: kworker/3:2 Tainted: P W O 5.4.38-gentoo-host #1

[317321.525494] RIP: e030:unmap_page_range+0x495/0xaa0

[317321.525598] Hardware name: Supermicro Super Server/X10DRi-T4+, BIOS 3.1 06/08/2018

[317321.525676] Code: 89 ee e8 be f9 ff ff 48 83 7c 24 30 00 49 89 c7 74 09 48 85 c0 0f 85 2e 04 00 00 41 f6 44 24 20 01 0f 84 6b 03 00 00 4c 8b 2b <48> c7 03 00 00 00 00 49 39 6c 24 10 48 89 e8 49 0f 46 44 24 10 4d

[317321.525748] Workqueue: events netstamp_clear

[317321.525808] RSP: e02b:ffffc9000a00bb58 EFLAGS: 00010202

[317321.526060] RIP: e030:xen_mc_flush+0x194/0x1c0

[317321.526095] RAX: ffffea001a1adf40 RBX: ffff888510ebd0e0 RCX: 0000000000000021

[317321.526098] RDX: 0000000000000000 RSI: 0000559801a1c000 RDI: 0000003038152125

[317321.526174] Code: 05 00 10 00 81 e8 ec 13 be 00 48 89 c1 48 89 45 18 48 c1 e9 3f 48 89 ce e9 03 ff ff ff 48 c7 45 18 ea ff ff ff be 01 00 00 00 <0f> 0b 8b 55 00 48 c7 c7 a0 8a fb 81 31 db 65 8b 0d e7 0a ff 7e e8

[317321.526211] RBP: 0000559801a1c000 R08: ffff8884ee5fd190 R09: 00000000003b1ad4

[317321.526214] R10: 00000000000279f0 R11: ffffea0012f16700 R12: ffffc9000a00bc90

[317321.526316] RSP: e02b:ffffc90004097d08 EFLAGS: 00010002

[317321.526385] R13: 0000003038152125 R14: 0000559801a1d000 R15: ffffea001a1adf40

[317321.526643] RAX: ffffffffffffffea RBX: 000000059ca3e000 RCX: 0000000000000001

[317321.526718] FS: 00007f5ddb094b80(0000) GS:ffff888686640000(0000) knlGS:0000000000000000

[317321.526796] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8886866d5b10

[317321.526855] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033

[317321.526858] CR2: ffff888510ebd0e0 CR3: 0000000566956000 CR4: 0000000000040660

[317321.526919] RBP: ffff8886866d5300 R08: 0000000000000000 R09: 00000000000007e1

[317321.526922] R10: 0000000000007ff0 R11: 0000000000000001 R12: 0000000000000000

[317321.526991] Call Trace:

[317321.527052] R13: 0000000000000020 R14: ffff88867a04b740 R15: 0000000000000001

[317321.527069] FS: 00007f5ddb094b80(0000) GS:ffff8886866c0000(0000) knlGS:0000000000000000

[317321.527122] unmap_vmas+0x7a/0xf0

[317321.527167] CS: 10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033

[317321.527170] CR2: 0000559801ac9f64 CR3: 000000059ca3e000 CR4: 0000000000040660

[317321.527232] exit_mmap+0xad/0x1a0

[317321.527288] Call Trace:

[317321.527349] mmput+0x52/0x110

[317321.527379] xen_write_cr3+0xf0/0x160

[317321.527437] flush_old_exec+0x261/0x680

[317321.527498] load_new_mm_cr3+0x39/0xc0

[317321.527529] load_elf_binary+0x31f/0x16a0

[317321.529744] switch_mm_irqs_off+0x199/0x4f0

[317321.532342] ? get_user_pages_remote+0x142/0x230

[317321.534662] ? __netif_receive_skb_core+0x3a/0xc70

[317321.536934] ? _cond_resched+0x15/0x30

[317321.539106] __text_poke+0x20e/0x4a0

[317321.540635] ? selinux_inode_permission+0x107/0x1d0

[317321.541836] text_poke_bp_batch+0xab/0x150

[317321.543033] ? load_misc_binary+0x4c/0x410 [binfmt_misc]

[317321.544213] arch_jump_label_transform_apply+0x2e/0x50

[317321.545389] search_binary_handler+0x84/0x1b0

[317321.546552] static_key_disable_cpuslocked+0x59/0x90

[317321.547683] __do_execve_file.isra.0+0x502/0x870

[317321.548784] static_key_disable+0x16/0x20

[317321.549843] __x64_sys_execve+0x35/0x40

[317321.550897] process_one_work+0x1e5/0x3a0

[317321.551906] do_syscall_64+0x5b/0x1a0

[317321.552894] worker_thread+0x4d/0x3d0

[317321.553880] entry_SYSCALL_64_after_hwframe+0x44/0xa9

[317321.554822] kthread+0xfb/0x130

[317321.555755] RIP: 0033:0x7f5ddb19af87

[317321.556674] ? process_one_work+0x3a0/0x3a0

[317321.557588] Code: Bad RIP value.

[317321.558471] ? kthread_park+0x90/0x90

[317321.559339] RSP: 002b:00007ffccf1b4ea8 EFLAGS: 00000202 ORIG_RAX: 000000000000003b

[317321.560172] ret_from_fork+0x35/0x40

[317321.560997] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f5ddb19af87

[317321.560998] RDX: 0000559802a26ff0 RSI: 0000559802a276d0 RDI: 0000559802a26f50

[317321.561825] ---[ end trace 1ddd52a17333fada ]---

[317321.562687] RBP: 0000559802a26f50 R08: 0000559802a276d0 R09: 0000559802a28270

[317321.563617] 1 of 1 multicall(s) failed: cpu 3

[317321.564472] R10: 00007f5ddb3429b0 R11: 0000000000000202 R12: 00000000ffffffff

[317321.565277] call 1: op=26 arg=[ffff8886866d5b10] result=-22

[317321.566039] R13: 0000559802a276d0 R14: 0000559802a26ff0 R15: 0000559802a26dd0

[317321.567053] ------------[ cut here ]------------

[317321.567843] Modules linked in: iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd auth_rpcgss nfs_acl lockd grace sunrpc br_netfilter xt_physdev xen_acpi_processor xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn xenfs xen_privcmd bridge 8021q garp mrp stp llc bonding intel_rapl_msr iTCO_wdt iTCO_vendor_support intel_rapl_common sb_edac intel_powerclamp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel intel_rapl_perf pcspkr i2c_i801 ast drm_vram_helper ttm i2c_algo_bit drm_kms_helper drm lpc_ich mei_me mei ipmi_ssif ixgbe mdio ptp ioatdma pps_core dca ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter binfmt_misc tun zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlua(PO)

[317321.568724] WARNING: CPU: 3 PID: 2162 at arch/x86/xen/multicalls.c:102 xen_mc_flush+0x194/0x1c0

[317321.569702] CR2: ffff888510ebd0e0

[317321.570470] Modules linked in: iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd auth_rpcgss nfs_acl lockd grace sunrpc br_netfilter xt_physdev xen_acpi_processor xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn xenfs xen_privcmd bridge 8021q garp mrp stp llc bonding intel_rapl_msr iTCO_wdt iTCO_vendor_support intel_rapl_common sb_edac intel_powerclamp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel intel_rapl_perf pcspkr i2c_i801 ast drm_vram_helper ttm i2c_algo_bit drm_kms_helper drm lpc_ich mei_me mei ipmi_ssif ixgbe mdio ptp ioatdma pps_core dca ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter binfmt_misc tun zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlua(PO)

[317321.571250] ---[ end trace 1ddd52a17333fadb ]---

[317321.572016] CPU: 3 PID: 2162 Comm: kworker/3:2 Tainted: P D W O 5.4.38-gentoo-host #1

[317321.641436] RIP: e030:unmap_page_range+0x495/0xaa0

[317321.641977] Hardware name: Supermicro Super Server/X10DRi-T4+, BIOS 3.1 06/08/2018

[317321.642997] Code: 89 ee e8 be f9 ff ff 48 83 7c 24 30 00 49 89 c7 74 09 48 85 c0 0f 85 2e 04 00 00 41 f6 44 24 20 01 0f 84 6b 03 00 00 4c 8b 2b <48> c7 03 00 00 00 00 49 39 6c 24 10 48 89 e8 49 0f 46 44 24 10 4d

[317321.649516] Workqueue: events netstamp_clear

[317321.650716] RSP: e02b:ffffc9000a00bb58 EFLAGS: 00010202

[317321.651897] RIP: e030:xen_mc_flush+0x194/0x1c0

[317321.653085] RAX: ffffea001a1adf40 RBX: ffff888510ebd0e0 RCX: 0000000000000021

[317321.654297] Code: 05 00 10 00 81 e8 ec 13 be 00 48 89 c1 48 89 45 18 48 c1 e9 3f 48 89 ce e9 03 ff ff ff 48 c7 45 18 ea ff ff ff be 01 00 00 00 <0f> 0b 8b 55 00 48 c7 c7 a0 8a fb 81 31 db 65 8b 0d e7 0a ff 7e e8

[317321.656804] RDX: 0000000000000000 RSI: 0000559801a1c000 RDI: 0000003038152125

[317321.656806] RBP: 0000559801a1c000 R08: ffff8884ee5fd190 R09: 00000000003b1ad4

[317321.658078] RSP: e02b:ffffc90004097d08 EFLAGS: 00010002

[317321.659351] R10: 00000000000279f0 R11: ffffea0012f16700 R12: ffffc9000a00bc90

[317321.660606] RAX: ffffffffffffffea RBX: 000000059ca3e000 RCX: 0000000000000001

[317321.661877] R13: 0000003038152125 R14: 0000559801a1d000 R15: ffffea001a1adf40

[317321.664593] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8886866d5b10

[317321.664595] RBP: ffff8886866d5300 R08: 0000000000000000 R09: 0000000000000008

[317321.665950] FS: 00007f5ddb094b80(0000) GS:ffff888686640000(0000) knlGS:0000000000000000

[317321.667281] R10: 0000000000007ff0 R11: ffffffff81e94060 R12: 0000000000000000

[317321.668625] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033

[317321.669984] R13: 0000000000000020 R14: ffff88867a04b740 R15: 0000000000000001

[317321.671356] CR2: 00007f5ddb19af5d CR3: 0000000566956000 CR4: 0000000000040660

[317321.672731] FS: 00007f5ddb094b80(0000) GS:ffff8886866c0000(0000) knlGS:0000000000000000

[317321.674252] ------------[ cut here ]------------

[317321.675581] CS: 10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033

[317321.675582] CR2: 0000559801ac9f64 CR3: 000000059ca3e000 CR4: 0000000000040660

[317321.675587] Call Trace:

[317321.675597] xen_write_cr3+0xf0/0x160

[317321.675604] load_new_mm_cr3+0x39/0xc0

[317321.681747] kernel BUG at arch/x86/kernel/paravirt.c:216!

[317321.683254] switch_mm_irqs_off+0x199/0x4f0

[317321.683258] ? netif_receive_skb_internal+0x1e/0xb0

[317321.683279] __text_poke+0x20e/0x4a0

[317321.683282] text_poke_bp_batch+0xab/0x150

[317321.683285] arch_jump_label_transform_apply+0x2e/0x50

[317321.691989] invalid opcode: 0000 [#2] SMP NOPTI

[317321.692508] static_key_disable_cpuslocked+0x59/0x90

[317321.695466] CPU: 0 PID: 25465 Comm: zfs_stats_cache Tainted: P D W O 5.4.38-gentoo-host #1

[317321.696945] static_key_disable+0x16/0x20

[317321.696950] process_one_work+0x1e5/0x3a0

[317321.703037] Hardware name: Supermicro Super Server/X10DRi-T4+, BIOS 3.1 06/08/2018

[317321.704565] worker_thread+0x4d/0x3d0

[317321.710655] RIP: e030:paravirt_enter_lazy_mmu+0x1c/0x20

[317321.710658] Code: 82 e9 c8 1d 07 00 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 65 8b 05 5c b7 fb 7e 85 c0 75 0c 65 c7 05 4d b7 fb 7e 01 00 00 00 c3 <0f> 0b 66 90 0f 1f 44 00 00 65 8b 05 3c b7 fb 7e 83 f8 01 75 0c 65

[317321.712081] kthread+0xfb/0x130

[317321.715023] RSP: e02b:ffffc9000a00bcf8 EFLAGS: 00010202

[317321.715026] RAX: 0000000000000001 RBX: ffff888658e99ff0 RCX: 000fffffffe00000

[317321.716398] ? process_one_work+0x3a0/0x3a0

[317321.716400] ? kthread_park+0x90/0x90

[317321.716406] ret_from_fork+0x35/0x40

[317321.716409] ---[ end trace 1ddd52a17333fadc ]---

[317321.721938] RDX: 0000000000000001 RSI: ffffea001963a640 RDI: ffff88867a04cc80

[317321.721940] RBP: ffffffffffffffff R08: 0000000000000000 R09: 00000000000037af

[317321.723352] 1 of 1 multicall(s) failed: cpu 3

[317321.729006] R10: 0000000000000000 R11: 0000000000000001 R12: ffffc9000a00be38

[317321.729008] R13: 00007fffffffe000 R14: 00007fffffffe000 R15: ffff88862f44aff8

[317321.730336] call 1: op=26 arg=[ffff8886866d5b10] result=-22

[317321.732821] FS: 00007f5ddb094b80(0000) GS:ffff888686600000(0000) knlGS:0000000000000000

[317321.734219] ------------[ cut here ]------------

[317321.739079] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033

[317321.739081] CR2: 00007fc50c030158 CR3: 000000000220a000 CR4: 0000000000040660

[317321.740343] WARNING: CPU: 3 PID: 2162 at arch/x86/xen/multicalls.c:102 xen_mc_flush+0x194/0x1c0

[317321.740345] Modules linked in: iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd auth_rpcgss nfs_acl lockd grace sunrpc br_netfilter xt_physdev xen_acpi_processor xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn xenfs xen_privcmd bridge 8021q garp mrp stp llc bonding intel_rapl_msr iTCO_wdt iTCO_vendor_support intel_rapl_common sb_edac intel_powerclamp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel intel_rapl_perf pcspkr i2c_i801 ast drm_vram_helper ttm i2c_algo_bit drm_kms_helper drm lpc_ich mei_me mei ipmi_ssif ixgbe mdio ptp ioatdma pps_core dca ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter binfmt_misc tun zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlua(PO)

[317321.745273] Call Trace:

[317321.746434] CPU: 3 PID: 2162 Comm: kworker/3:2 Tainted: P D W O 5.4.38-gentoo-host #1

[317321.746435] Hardware name: Supermicro Super Server/X10DRi-T4+, BIOS 3.1 06/08/2018

[317321.746441] Workqueue: events netstamp_clear

[317321.746444] RIP: e030:xen_mc_flush+0x194/0x1c0

[317321.750741] unmap_page_range+0x44c/0xaa0

[317321.751816] Code: 05 00 10 00 81 e8 ec 13 be 00 48 89 c1 48 89 45 18 48 c1 e9 3f 48 89 ce e9 03 ff ff ff 48 c7 45 18 ea ff ff ff be 01 00 00 00 <0f> 0b 8b 55 00 48 c7 c7 a0 8a fb 81 31 db 65 8b 0d e7 0a ff 7e e8

[317321.754758] unmap_vmas+0x7a/0xf0

[317321.757083] RSP: e02b:ffffc90004097d08 EFLAGS: 00010002

[317321.757084] RAX: ffffffffffffffea RBX: 000000059ca3e000 RCX: 0000000000000001

[317321.757085] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8886866d5b10

[317321.757086] RBP: ffff8886866d5300 R08: 0000000000000000 R09: 0000000000000005

[317321.757086] R10: 0000000000007ff0 R11: ffffffff81e94060 R12: 0000000000000000

[317321.757087] R13: 0000000000000020 R14: ffff88867a04b740 R15: 0000000000000001

[317321.757097] FS: 00007f5ddb094b80(0000) GS:ffff8886866c0000(0000) knlGS:0000000000000000

[317321.761641] exit_mmap+0xad/0x1a0

[317321.762719] CS: 10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033

[317321.762720] CR2: 0000559801ac9f64 CR3: 000000059ca3e000 CR4: 0000000000040660

[317321.762725] Call Trace:

[317321.762729] xen_write_cr3+0xf0/0x160

[317321.762734] load_new_mm_cr3+0x39/0xc0

[317321.767637] mmput+0x52/0x110

[317321.768765] switch_mm_irqs_off+0x199/0x4f0

[317321.771139] do_exit+0x304/0xac0

[317321.772116] ? netif_receive_skb_list_internal+0x49/0x2c0

[317321.776094] ? __x64_sys_execve+0x35/0x40

[317321.777184] __text_poke+0x20e/0x4a0

[317321.781732] rewind_stack_do_exit+0x17/0x20

[317321.782831] text_poke_bp_batch+0xab/0x150

[317321.782834] arch_jump_label_transform_apply+0x2e/0x50

[317321.787178] Modules linked in: iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd auth_rpcgss nfs_acl lockd grace sunrpc br_netfilter xt_physdev xen_acpi_processor xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn xenfs xen_privcmd bridge 8021q garp mrp stp llc bonding intel_rapl_msr iTCO_wdt iTCO_vendor_support intel_rapl_common sb_edac intel_powerclamp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel intel_rapl_perf pcspkr i2c_i801 ast drm_vram_helper ttm i2c_algo_bit drm_kms_helper drm lpc_ich mei_me mei ipmi_ssif ixgbe mdio ptp ioatdma pps_core dca ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter binfmt_misc tun zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlua(PO)

[317321.788176] static_key_disable_cpuslocked+0x59/0x90

[317321.790481] ---[ end trace 1ddd52a17333fadd ]---

[317321.791460] static_key_disable+0x16/0x20

[317321.791462] process_one_work+0x1e5/0x3a0

[317321.791464] worker_thread+0x4d/0x3d0

[317321.791466] kthread+0xfb/0x130

[317321.791468] ? process_one_work+0x3a0/0x3a0

[317321.791470] ? kthread_park+0x90/0x90

[317321.791473] ret_from_fork+0x35/0x40

[317321.791475] ---[ end trace 1ddd52a17333fade ]---

[317321.791478] 1 of 1 multicall(s) failed: cpu 3

[317321.868770] RIP: e030:unmap_page_range+0x495/0xaa0

[317321.869585] call 1: op=26 arg=[ffff8886866d5b10] result=-22

[317321.869762] ------------[ cut here ]------------

[317321.874624] Code: 89 ee e8 be f9 ff ff 48 83 7c 24 30 00 49 89 c7 74 09 48 85 c0 0f 85 2e 04 00 00 41 f6 44 24 20 01 0f 84 6b 03 00 00 4c 8b 2b <48> c7 03 00 00 00 00 49 39 6c 24 10 48 89 e8 49 0f 46 44 24 10 4d

[317321.875681] WARNING: CPU: 3 PID: 2162 at arch/x86/xen/multicalls.c:102 xen_mc_flush+0x194/0x1c0

[317321.875682] Modules linked in: iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd auth_rpcgss nfs_acl lockd grace sunrpc br_netfilter xt_physdev xen_acpi_processor xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn xenfs xen_privcmd bridge 8021q garp mrp stp llc bonding intel_rapl_msr iTCO_wdt iTCO_vendor_support intel_rapl_common sb_edac intel_powerclamp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel intel_rapl_perf pcspkr i2c_i801 ast drm_vram_helper ttm i2c_algo_bit drm_kms_helper drm lpc_ich mei_me mei ipmi_ssif ixgbe mdio ptp ioatdma pps_core dca ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter binfmt_misc tun zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlua(PO)

[317321.875704] CPU: 3 PID: 2162 Comm: kworker/3:2 Tainted: P D W O 5.4.38-gentoo-host #1

[317321.875704] Hardware name: Supermicro Super Server/X10DRi-T4+, BIOS 3.1 06/08/2018

[317321.875721] Workqueue: events netstamp_clear

[317321.875723] RIP: e030:xen_mc_flush+0x194/0x1c0

[317321.875724] Code: 05 00 10 00 81 e8 ec 13 be 00 48 89 c1 48 89 45 18 48 c1 e9 3f 48 89 ce e9 03 ff ff ff 48 c7 45 18 ea ff ff ff be 01 00 00 00 <0f> 0b 8b 55 00 48 c7 c7 a0 8a fb 81 31 db 65 8b 0d e7 0a ff 7e e8

[317321.875725] RSP: e02b:ffffc90004097d08 EFLAGS: 00010002

[317321.875726] RAX: ffffffffffffffea RBX: 000000059ca3e000 RCX: 0000000000000001

[317321.875727] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8886866d5b10

[317321.875728] RBP: ffff8886866d5300 R08: 0000000000000000 R09: ffff88867ec01c38

[317321.875729] R10: 0000000000007ff0 R11: ffffffff8224c7a8 R12: 0000000000000000

[317321.875729] R13: 0000000000000020 R14: ffff88867a04b740 R15: 0000000000000001

[317321.875739] FS: 00007f5ddb094b80(0000) GS:ffff8886866c0000(0000) knlGS:0000000000000000

[317321.875740] CS: 10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033

[317321.875741] CR2: 0000559801ac9f64 CR3: 000000059ca3e000 CR4: 0000000000040660

[317321.875745] Call Trace:

[317321.875750] xen_write_cr3+0xf0/0x160

[317321.875753] load_new_mm_cr3+0x39/0xc0

[317321.875755] switch_mm_irqs_off+0x199/0x4f0

[317321.875757] ? dev_queue_xmit_nit+0xff/0x250

[317321.875759] __text_poke+0x20e/0x4a0

[317321.875762] text_poke_bp_batch+0xea/0x150

[317321.875764] arch_jump_label_transform_apply+0x2e/0x50

[317321.875767] static_key_disable_cpuslocked+0x59/0x90

[317321.875769] static_key_disable+0x16/0x20

[317321.875772] process_one_work+0x1e5/0x3a0

[317321.883271] RSP: e02b:ffffc9000a00bb58 EFLAGS: 00010202

[317321.884441] worker_thread+0x4d/0x3d0

[317321.884444] kthread+0xfb/0x130

[317321.884445] ? process_one_work+0x3a0/0x3a0

[317321.884447] ? kthread_park+0x90/0x90

[317321.884452] ret_from_fork+0x35/0x40

[317321.884454] ---[ end trace 1ddd52a17333fadf ]---

[317321.884456] 1 of 1 multicall(s) failed: cpu 3

[317321.884457] call 1: op=26 arg=[ffff8886866d5b10] result=-22

[317321.884566] ------------[ cut here ]------------

[317321.884568] WARNING: CPU: 3 PID: 2162 at arch/x86/xen/multicalls.c:102 xen_mc_flush+0x194/0x1c0

[317321.884569] Modules linked in: iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd auth_rpcgss nfs_acl lockd grace sunrpc br_netfilter xt_physdev xen_acpi_processor xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn xenfs xen_privcmd bridge 8021q garp mrp stp llc bonding intel_rapl_msr iTCO_wdt iTCO_vendor_support intel_rapl_common sb_edac intel_powerclamp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel intel_rapl_perf pcspkr i2c_i801 ast drm_vram_helper ttm i2c_algo_bit drm_kms_helper drm lpc_ich mei_me mei ipmi_ssif ixgbe mdio ptp ioatdma pps_core dca ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter binfmt_misc tun zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlua(PO)

[317321.889790] RAX: ffffea001a1adf40 RBX: ffff888510ebd0e0 RCX: 0000000000000021

[317321.890961] CPU: 3 PID: 2162 Comm: kworker/3:2 Tainted: P D W O 5.4.38-gentoo-host #1

[317321.890962] Hardware name: Supermicro Super Server/X10DRi-T4+, BIOS 3.1 06/08/2018

[317321.890965] Workqueue: events netstamp_clear

[317321.890967] RIP: e030:xen_mc_flush+0x194/0x1c0

[317321.890968] Code: 05 00 10 00 81 e8 ec 13 be 00 48 89 c1 48 89 45 18 48 c1 e9 3f 48 89 ce e9 03 ff ff ff 48 c7 45 18 ea ff ff ff be 01 00 00 00 <0f> 0b 8b 55 00 48 c7 c7 a0 8a fb 81 31 db 65 8b 0d e7 0a ff 7e e8

[317321.890969] RSP: e02b:ffffc90004097d08 EFLAGS: 00010002

[317321.890970] RAX: ffffffffffffffea RBX: 000000059ca3e000 RCX: 0000000000000001

[317321.890971] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8886866d5b10

[317321.890972] RBP: ffff8886866d5300 R08: 0000000000000000 R09: ffffffff828b9af4

[317321.890972] R10: 0000000000007ff0 R11: 000000000001389c R12: 0000000000000000

[317321.890973] R13: 0000000000000020 R14: ffff88867a04b740 R15: 0000000000000001

[317321.890982] FS: 00007f5ddb094b80(0000) GS:ffff8886866c0000(0000) knlGS:0000000000000000

[317321.890982] CS: 10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033

[317321.890983] CR2: 0000559801ac9f64 CR3: 000000059ca3e000 CR4: 0000000000040660

[317321.890987] Call Trace:

[317321.890991] xen_write_cr3+0xf0/0x160

[317321.890993] load_new_mm_cr3+0x39/0xc0

[317321.890995] switch_mm_irqs_off+0x199/0x4f0

[317321.890996] ? netif_rx_internal+0x1e/0x100

[317321.890999] __text_poke+0x20e/0x4a0

[317321.891001] text_poke_bp_batch+0xea/0x150

[317321.891003] arch_jump_label_transform_apply+0x2e/0x50

[317321.891016] static_key_disable_cpuslocked+0x59/0x90

[317321.891018] static_key_disable+0x16/0x20

[317321.891020] process_one_work+0x1e5/0x3a0

[317321.891022] worker_thread+0x4d/0x3d0

[317321.891024] kthread+0xfb/0x130

[317321.891025] ? process_one_work+0x3a0/0x3a0

[317321.891027] ? kthread_park+0x90/0x90

[317321.891029] ret_from_fork+0x35/0x40

[317321.891031] ---[ end trace 1ddd52a17333fae0 ]---

[317321.891033] 1 of 1 multicall(s) failed: cpu 3

[317321.891034] call 1: op=26 arg=[ffff8886866d5b10] result=-22

[317321.891148] ------------[ cut here ]------------

[317321.896271] RDX: 0000000000000000 RSI: 0000559801a1c000 RDI: 0000003038152125

[317321.897459] WARNING: CPU: 3 PID: 2162 at arch/x86/xen/multicalls.c:102 xen_mc_flush+0x194/0x1c0

[317321.897460] Modules linked in: iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd auth_rpcgss nfs_acl lockd grace sunrpc br_netfilter xt_physdev xen_acpi_processor xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn xenfs xen_privcmd bridge 8021q garp mrp stp llc bonding intel_rapl_msr iTCO_wdt iTCO_vendor_support intel_rapl_common sb_edac intel_powerclamp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel intel_rapl_perf pcspkr i2c_i801 ast drm_vram_helper ttm i2c_algo_bit drm_kms_helper drm lpc_ich mei_me mei ipmi_ssif ixgbe mdio ptp ioatdma pps_core dca ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter binfmt_misc tun zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlua(PO)

[317321.897478] CPU: 3 PID: 2162 Comm: kworker/3:2 Tainted: P D W O 5.4.38-gentoo-host #1

[317321.897479] Hardware name: Supermicro Super Server/X10DRi-T4+, BIOS 3.1 06/08/2018

[317321.897481] Workqueue: events netstamp_clear

[317321.897483] RIP: e030:xen_mc_flush+0x194/0x1c0

[317321.897484] Code: 05 00 10 00 81 e8 ec 13 be 00 48 89 c1 48 89 45 18 48 c1 e9 3f 48 89 ce e9 03 ff ff ff 48 c7 45 18 ea ff ff ff be 01 00 00 00 <0f> 0b 8b 55 00 48 c7 c7 a0 8a fb 81 31 db 65 8b 0d e7 0a ff 7e e8

[317321.897485] RSP: e02b:ffffc90004097d08 EFLAGS: 00010002

[317321.897486] RAX: ffffffffffffffea RBX: 000000059ca3e000 RCX: 0000000000000001

[317321.897487] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8886866d5b10

[317321.897488] RBP: ffff8886866d5300 R08: 0000000000000000 R09: ffffffff828b9af4

[317321.897488] R10: 0000000000007ff0 R11: 0000000000012c70 R12: 0000000000000000

[317321.897489] R13: 0000000000000020 R14: ffff88867a04b740 R15: 0000000000000001

[317321.897498] FS: 00007f5ddb094b80(0000) GS:ffff8886866c0000(0000) knlGS:0000000000000000

[317321.897498] CS: 10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033

[317321.897499] CR2: 0000559801ac9f64 CR3: 000000059ca3e000 CR4: 0000000000040660

[317321.897503] Call Trace:

[317321.897506] xen_write_cr3+0xf0/0x160

[317321.897509] load_new_mm_cr3+0x39/0xc0

[317321.897511] switch_mm_irqs_off+0x199/0x4f0

[317321.897513] ? __netif_receive_skb_core+0x39/0xc70

[317321.897515] __text_poke+0x20e/0x4a0

[317321.897517] text_poke_bp_batch+0xea/0x150

[317321.897519] arch_jump_label_transform_apply+0x2e/0x50

[317321.897521] static_key_disable_cpuslocked+0x59/0x90

[317321.897523] static_key_disable+0x16/0x20

[317321.897524] process_one_work+0x1e5/0x3a0

[317321.897526] worker_thread+0x4d/0x3d0

[317321.897529] kthread+0xfb/0x130

[317321.901544] RBP: 0000559801a1c000 R08: ffff8884ee5fd190 R09: 00000000003b1ad4

[317321.902727] ? process_one_work+0x3a0/0x3a0

[317321.902729] ? kthread_park+0x90/0x90

[317321.902731] ret_from_fork+0x35/0x40

[317321.902734] ---[ end trace 1ddd52a17333fae1 ]---

[317321.906588] R10: 00000000000279f0 R11: ffffea0012f16700 R12: ffffc9000a00bc90

[317321.907713] 1 of 1 multicall(s) failed: cpu 3

[317321.907714] call 1: op=26 arg=[ffff8886866d5b10] result=-22

[317321.907833] ------------[ cut here ]------------

[317321.913095] R13: 0000003038152125 R14: 0000559801a1d000 R15: ffffea001a1adf40

[317321.914285] WARNING: CPU: 3 PID: 2162 at arch/x86/xen/multicalls.c:102 xen_mc_flush+0x194/0x1c0

[317321.914288] Modules linked in: iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd auth_rpcgss nfs_acl lockd grace sunrpc br_netfilter xt_physdev xen_acpi_processor xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn xenfs xen_privcmd bridge 8021q garp mrp stp llc bonding intel_rapl_msr iTCO_wdt iTCO_vendor_support intel_rapl_common sb_edac intel_powerclamp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel intel_rapl_perf pcspkr i2c_i801 ast drm_vram_helper ttm i2c_algo_bit drm_kms_helper drm lpc_ich mei_me mei ipmi_ssif ixgbe mdio ptp ioatdma pps_core dca ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter binfmt_misc tun zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlua(PO)

[317321.914306] CPU: 3 PID: 2162 Comm: kworker/3:2 Tainted: P D W O 5.4.38-gentoo-host #1

[317321.914306] Hardware name: Supermicro Super Server/X10DRi-T4+, BIOS 3.1 06/08/2018

[317321.914309] Workqueue: events netstamp_clear

[317321.914310] RIP: e030:xen_mc_flush+0x194/0x1c0

[317321.914312] Code: 05 00 10 00 81 e8 ec 13 be 00 48 89 c1 48 89 45 18 48 c1 e9 3f 48 89 ce e9 03 ff ff ff 48 c7 45 18 ea ff ff ff be 01 00 00 00 <0f> 0b 8b 55 00 48 c7 c7 a0 8a fb 81 31 db 65 8b 0d e7 0a ff 7e e8

[317321.914313] RSP: e02b:ffffc90004097d08 EFLAGS: 00010002

[317321.914314] RAX: ffffffffffffffea RBX: 000000059ca3e000 RCX: 0000000000000001

[317321.914314] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8886866d5b10

[317321.914315] RBP: ffff8886866d5300 R08: 0000000000000000 R09: ffffffff828b9af4

[317321.914316] R10: 0000000000007ff0 R11: 0000000000011f94 R12: 0000000000000000

[317321.914317] R13: 0000000000000020 R14: ffff88867a04b740 R15: 0000000000000001

[317321.914329] FS: 00007f5ddb094b80(0000) GS:ffff8886866c0000(0000) knlGS:0000000000000000

[317321.914330] CS: 10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033

[317321.914331] CR2: 0000559801ac9f64 CR3: 000000059ca3e000 CR4: 0000000000040660

[317321.914337] Call Trace:

[317321.919667] FS: 00007f5ddb094b80(0000) GS:ffff888686600000(0000) knlGS:0000000000000000

[317321.920718] xen_write_cr3+0xf0/0x160

[317321.920721] load_new_mm_cr3+0x39/0xc0

[317321.920723] switch_mm_irqs_off+0x199/0x4f0

[317321.920725] ? netif_receive_skb_internal+0x1d/0xb0

[317321.920728] __text_poke+0x20e/0x4a0

[317321.922799] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033

[317321.923954] text_poke_bp_batch+0xea/0x150

[317321.923956] arch_jump_label_transform_apply+0x2e/0x50

[317321.923958] static_key_disable_cpuslocked+0x59/0x90

[317321.923960] static_key_disable+0x16/0x20

[317321.923962] process_one_work+0x1e5/0x3a0

[317321.923963] worker_thread+0x4d/0x3d0

[317321.923966] kthread+0xfb/0x130

[317321.923967] ? process_one_work+0x3a0/0x3a0

[317321.923968] ? kthread_park+0x90/0x90

[317321.923971] ret_from_fork+0x35/0x40

[317321.923973] ---[ end trace 1ddd52a17333fae2 ]---

[317321.923975] 1 of 1 multicall(s) failed: cpu 3

[317321.923975] call 1: op=26 arg=[ffff8886866d5b10] result=-22

[317321.924078] ------------[ cut here ]------------

[317321.929186] CR2: 00007fc50c030158 CR3: 000000000220a000 CR4: 0000000000040660

[317321.930247] WARNING: CPU: 3 PID: 2162 at arch/x86/xen/multicalls.c:102 xen_mc_flush+0x194/0x1c0

[317321.930247] Modules linked in: iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd auth_rpcgss nfs_acl lockd grace sunrpc br_netfilter xt_physdev xen_acpi_processor xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn xenfs xen_privcmd bridge 8021q garp mrp stp llc bonding intel_rapl_msr iTCO_wdt iTCO_vendor_support intel_rapl_common sb_edac intel_powerclamp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel intel_rapl_perf pcspkr i2c_i801 ast drm_vram_helper ttm i2c_algo_bit drm_kms_helper drm lpc_ich mei_me mei ipmi_ssif ixgbe mdio ptp ioatdma pps_core dca ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter binfmt_misc tun zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlua(PO)

[317321.930265] CPU: 3 PID: 2162 Comm: kworker/3:2 Tainted: P D W O 5.4.38-gentoo-host #1

[317321.930265] Hardware name: Supermicro Super Server/X10DRi-T4+, BIOS 3.1 06/08/2018

[317321.930268] Workqueue: events netstamp_clear

[317321.930269] RIP: e030:xen_mc_flush+0x194/0x1c0

[317321.930270] Code: 05 00 10 00 81 e8 ec 13 be 00 48 89 c1 48 89 45 18 48 c1 e9 3f 48 89 ce e9 03 ff ff ff 48 c7 45 18 ea ff ff ff be 01 00 00 00 <0f> 0b 8b 55 00 48 c7 c7 a0 8a fb 81 31 db 65 8b 0d e7 0a ff 7e e8

[317321.930271] RSP: e02b:ffffc90004097d08 EFLAGS: 00010002

[317321.930272] RAX: ffffffffffffffea RBX: 000000059ca3e000 RCX: 0000000000000001

[317321.930273] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8886866d5b10

[317321.930274] RBP: ffff8886866d5300 R08: 0000000000000000 R09: ffffffff828b9af4

[317321.930274] R10: 0000000000007ff0 R11: 00000000000112c0 R12: 0000000000000000

[317321.930275] R13: 0000000000000020 R14: ffff88867a04b740 R15: 0000000000000001

[317321.930283] FS: 00007f5ddb094b80(0000) GS:ffff8886866c0000(0000) knlGS:0000000000000000

[317321.930284] CS: 10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033

[317321.930285] CR2: 0000559801ac9f64 CR3: 000000059ca3e000 CR4: 0000000000040660

[317321.930289] Call Trace:

[317321.930292] xen_write_cr3+0xf0/0x160

[317321.930295] load_new_mm_cr3+0x39/0xc0

[317321.930297] switch_mm_irqs_off+0x199/0x4f0

[317321.930298] ? netif_receive_skb_list_internal+0x48/0x2c0

[317321.930301] __text_poke+0x20e/0x4a0

[317321.930303] text_poke_bp_batch+0xea/0x150

[317321.930305] arch_jump_label_transform_apply+0x2e/0x50

[317321.930307] static_key_disable_cpuslocked+0x59/0x90

[317321.930309] static_key_disable+0x16/0x20

[317321.930311] process_one_work+0x1e5/0x3a0

[317321.930312] worker_thread+0x4d/0x3d0

[317321.930314] kthread+0xfb/0x130

[317321.930316] ? process_one_work+0x3a0/0x3a0

[317321.930318] ? kthread_park+0x90/0x90

[317321.930321] ret_from_fork+0x35/0x40

[317321.930324] ---[ end trace 1ddd52a17333fae3 ]---

[317321.935393] Fixing recursive fault but reboot is needed!

[317321.936350] 1 of 1 multicall(s) failed: cpu 3

[317321.936351] call 1: op=26 arg=[ffff8886866d5b10] result=-22

[558278.124770] connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4852929746, last ping 4852935168, now 4852940288

[558278.129353] connection1:0: detected conn error (1022)