Re: [BUG] Xen BUG at mce_amd.c:100 (mc_ec2type) on Zen 3; host left wedged, no reboot

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 23/10/2026 8:32 am, Peter Deggendorfer wrote:
> Hey Xen Team,
>
> I'm hitting a reproducible hypervisor panic in the AMD non-fatal MCE
> polling path on a Ryzen 9 5950X (Zen 3). Eight occurrences since
> 2026-09-09, between 12 hours and 2 weeks apart, always CPU 2.
>
> Environment
> -----------
> Xen:   4.20.4-pre (Debian 4.20.3+127-gc42374a105-0+deb13u1), and still
>        present in 4.20.5-pre (Debian 4.20.4+39-gf1e1321c87-0+deb13u1)
> dom0:  Linux 6.12.111+deb13-amd64 (Debian trixie), PV, 32 vCPUs
> CPU:   AMD Family 25 (0x19) Model 33 (0x21) Stepping 2, Ryzen 9 5950X,
>        microcode 0x0a201210
> Cmdline: dom0_mem=4096M,max:4096M ucode=scan console_timestamps=datems
>
> The panic
> ---------
> (XEN) [2026-10-19 08:58:18.727] Xen BUG at
> arch/x86/cpu/mcheck/mce_amd.c:100
> (XEN) CPU:    2
> (XEN) RIP:    e008:[<ffff82d04028a650>]
> mce_amd.c#mc_ec2type.part.0+0/0x10
> (XEN) rbx: 0000000000000445   r12: 0000000000000011   r15:
> 0000000000000446
> (XEN) Xen code around <ffff82d04028a650>:
> (XEN)  90 90 90 90 90 90 90 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90
> 90 90
> (XEN) Xen call trace:
> (XEN)    [<ffff82d04028a650>] R mce_amd.c#mc_ec2type.part.0+0/0x10
> (XEN)    [<ffff82d04028a699>] S mc_amd_recoverable_scan+0x39/0x40
> (XEN)    [<ffff82d04028cfdd>] S mcheck_mca_logout+0x42d/0x840
> (XEN)    [<ffff82d04028a35c>] S
> amd_nonfatal.c#mce_amd_checkregs+0x1c/0x100
> (XEN)    [<ffff82d040240e8c>] S smp_call_function_interrupt+0x4c/0x90
> (XEN)    [<ffff82d0403123d5>] S do_IRQ+0x2a5/0x650
> (XEN)    [<ffff82d040201915>] S common_interrupt+0x135/0x150
> (XEN)    [<ffff82d0402883ca>] S cpu_idle.c#acpi_idle_do_entry+0x8a/0x140
> (XEN)    [<ffff82d0402887cf>] S
> cpu_idle.c#acpi_processor_idle+0x1ef/0x650
> (XEN)    [<ffff82d040302cf1>] S domain.c#idle_loop+0xc1/0x130
> (XEN) Panic on CPU 2
>
> The outlined mc_ec2type.part.0 contains nothing but the ud2, so this is
> the "unreachable" default branch of mc_ec2type() -- the extended error
> code matches none of MC_EC_TLB_TYPE, MC_EC_MEM_TYPE, MC_EC_BUS_TYPE.

I think it's safe to just drop the BUG().

That code is literally worse than useless, because if nothing else it's
just hiding the diagnostics.

I guess it means AMD have added new statuses.

>
> Across the eight occurrences, r12 appears to be the bank index from the
> loop in mcheck_mca_logout(), with rbx == 0x401 + 4 * r12 (MCi_STATUS)
> and r15 == rbx + 1. Three different banks are involved: 10, 17 and 23.
>
> Not a hardware defect
> ---------------------
> - Started within days of upgrading two hosts from Debian bookworm
>   (Xen 4.17) to trixie (Xen 4.20). They had run 3 years on 4.17 with no
>   incident.
> - Three different banks, so not a single failing block.
> - Updating to the current Debian package (4.20.5-pre, built 2026-10-16)
>   changed nothing.
>
> Possibly related: 532988dccc19 ("x86/mcheck: fix mca_allbanks
> updating").
>
> Also possibly related on the Linux side: 0422b07bc4c2 ("x86/mce/amd:
> Filter bogus hardware errors on Zen3 clients"), where bogus MCA values
> on Zen 3 are filtered rather than treated as errors.
> https://ratatoskr.run/stable/2026/04/7143905

>From above, you're on a Vermeer, not a Cezanne, so I don't think that
Linux patch is relevant.

However, your observation about "fix mca_allbanks updating" quite
possibly is.  That was backported into Xen 4.20.1 as 247b708b1400a, and
Debian has 4.20.3

Are you able to experiment with that patch reverted?

Alternatively, are you able to experiment with the BUG() removed?

>
> Diagnostics / workaround
> ------------------------
> mce_verbosity=verbose yields nothing useful.
> max_cstate=1 made no difference. mce=false is the only thing that stops
> it, and it is what I'm running now.

The fact that mce=false isn't causing machine resets means it's clearly
something the hardware thinks isn't fatal.

But, I am concerned by the fact it's always on CPU 2, and multiple
banks.  Is this true for both systems?

>
> One more thing: the host does not actually reboot after the panic.
> "Reboot in five seconds" is printed, but the ring buffer continues with
> normal activity minutes later and the remaining 31 CPUs keep running.
> CPU 2 stays wedged, and "xl debug-keys t" then hangs. Afterwards dom0
> and guest vCPUs start losing timer interrupts permanently (RCU stalls
> with a frozen timer-softirq count while dynticks keeps advancing), so
> VMs hang silently hours or days later.

I've recently noticed that crashing with interrupts off doesn't reboot
any more.  I thought it was something local to my development, but
apparently not if you're seeing the same.

>
>
> I've attached the full xl dmesg log of the last crash, and I'm happy
> to provide more if needed. I can also test patches.
>
> Thanks for your work, and thanks in advance for taking a look.
>
> Peter 

Thankyou,

~Andrew