Re: [BUG] Xen BUG at mce_amd.c:100 (mc_ec2type) on Zen 3; host left wedged, no reboot

[Peter Deggendorfer <pd@xxxxxxxxxxxxxxx>]

Hi Andrew,

Thanks a lot for the quick response.


> On 23/10/2026 8:32 am, Peter Deggendorfer wrote:
> > Hey Xen Team,
> >
> > I'm hitting a reproducible hypervisor panic in the AMD non-fatal MCE
> > polling path on a Ryzen 9 5950X (Zen 3). Eight occurrences since
> > 2026-09-09, between 12 hours and 2 weeks apart, always CPU 2.
> >
> > Environment
> > -----------
> > Xen:   4.20.4-pre (Debian 4.20.3+127-gc42374a105-0+deb13u1), and still
> >         present in 4.20.5-pre (Debian 4.20.4+39-gf1e1321c87-0+deb13u1)
> > dom0:  Linux 6.12.111+deb13-amd64 (Debian trixie), PV, 32 vCPUs
> > CPU:   AMD Family 25 (0x19) Model 33 (0x21) Stepping 2, Ryzen 9 5950X,
> >         microcode 0x0a201210
> > Cmdline: dom0_mem=4096M,max:4096M ucode=scan console_timestamps=datems
> >
> > The panic
> > ---------
> > (XEN) [2026-10-19 08:58:18.727] Xen BUG at
> > arch/x86/cpu/mcheck/mce_amd.c:100
> > (XEN) CPU:    2
> > (XEN) RIP:    e008:[<ffff82d04028a650>]
> > mce_amd.c#mc_ec2type.part.0+0/0x10
> > (XEN) rbx: 0000000000000445   r12: 0000000000000011   r15:
> > 0000000000000446
> > (XEN) Xen code around <ffff82d04028a650>:
> > (XEN)  90 90 90 90 90 90 90 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90
> > 90 90
> > (XEN) Xen call trace:
> > (XEN)    [<ffff82d04028a650>] R mce_amd.c#mc_ec2type.part.0+0/0x10
> > (XEN)    [<ffff82d04028a699>] S mc_amd_recoverable_scan+0x39/0x40
> > (XEN)    [<ffff82d04028cfdd>] S mcheck_mca_logout+0x42d/0x840
> > (XEN)    [<ffff82d04028a35c>] S
> > amd_nonfatal.c#mce_amd_checkregs+0x1c/0x100
> > (XEN)    [<ffff82d040240e8c>] S smp_call_function_interrupt+0x4c/0x90
> > (XEN)    [<ffff82d0403123d5>] S do_IRQ+0x2a5/0x650
> > (XEN)    [<ffff82d040201915>] S common_interrupt+0x135/0x150
> > (XEN)    [<ffff82d0402883ca>] S cpu_idle.c#acpi_idle_do_entry+0x8a/0x140
> > (XEN)    [<ffff82d0402887cf>] S
> > cpu_idle.c#acpi_processor_idle+0x1ef/0x650
> > (XEN)    [<ffff82d040302cf1>] S domain.c#idle_loop+0xc1/0x130
> > (XEN) Panic on CPU 2
> >
> > The outlined mc_ec2type.part.0 contains nothing but the ud2, so this is
> > the "unreachable" default branch of mc_ec2type() -- the extended error
> > code matches none of MC_EC_TLB_TYPE, MC_EC_MEM_TYPE, MC_EC_BUS_TYPE.
> I think it's safe to just drop the BUG().
>
> That code is literally worse than useless, because if nothing else it's
> just hiding the diagnostics.
>
> I guess it means AMD have added new statuses.
>
> > Across the eight occurrences, r12 appears to be the bank index from the
> > loop in mcheck_mca_logout(), with rbx == 0x401 + 4 * r12 (MCi_STATUS)
> > and r15 == rbx + 1. Three different banks are involved: 10, 17 and 23.
> >
> > Not a hardware defect
> > ---------------------
> > - Started within days of upgrading two hosts from Debian bookworm
> >    (Xen 4.17) to trixie (Xen 4.20). They had run 3 years on 4.17 with no
> >    incident.
> > - Three different banks, so not a single failing block.
> > - Updating to the current Debian package (4.20.5-pre, built 2026-10-16)
> >    changed nothing.
> >
> > Possibly related: 532988dccc19 ("x86/mcheck: fix mca_allbanks
> > updating").
> >
> > Also possibly related on the Linux side: 0422b07bc4c2 ("x86/mce/amd:
> > Filter bogus hardware errors on Zen3 clients"), where bogus MCA values
> > on Zen 3 are filtered rather than treated as errors.
> > https://ratatoskr.run/stable/2026/04/7143905
>  From above, you're on a Vermeer, not a Cezanne, so I don't think that
> Linux patch is relevant.
>
> However, your observation about "fix mca_allbanks updating" quite
> possibly is.  That was backported into Xen 4.20.1 as 247b708b1400a, and
> Debian has 4.20.3
>
> Are you able to experiment with that patch reverted?
>
> Alternatively, are you able to experiment with the BUG() removed?
I went with dropping the BUG() rather than the revert.
Reverting 247b708b1400a should stop any bank from being read at
all, which is effectively what mce=false already does here, so I don't
expect it to tell us anything new. I'll run it too if you'd like.

I'm not familiar with the Xen codebase, but I've built a package with
mc_ec2type() changed to:

    if ( errorcode & MC_EC_TLB_TYPE )
        return MC_EC_TLB_TYPE;

    printk(XENLOG_WARNING "MCE: unknown AMD error code %#x\n", errorcode);

    return 0;


Crashes have been 12 hours to 2 weeks apart, so it may take a while
before I can report back.


> > Diagnostics / workaround
> > ------------------------
> > mce_verbosity=verbose yields nothing useful.
> > max_cstate=1 made no difference. mce=false is the only thing that stops
> > it, and it is what I'm running now.
> The fact that mce=false isn't causing machine resets means it's clearly
> something the hardware thinks isn't fatal.
>
> But, I am concerned by the fact it's always on CPU 2, and multiple
> banks.  Is this true for both systems?
To clarify one thing from my report: it is actually a single host whose
complete hardware (board, CPU, RAM) was replaced on
2026-10-07, keeping only the disks and thus the installation. The
crashes happened both before and after that swap. 10 more hosts
with same hardware still run on bookworm (xen 4.17) for years now
without any incidents.

So yes, always CPU 2. The bank index varies: on the current hardware
I have seen banks 23, 17 and 10 across four crashes, and banks 23 and
17 on the previous hardware.

> > One more thing: the host does not actually reboot after the panic.
> > "Reboot in five seconds" is printed, but the ring buffer continues with
> > normal activity minutes later and the remaining 31 CPUs keep running.
> > CPU 2 stays wedged, and "xl debug-keys t" then hangs. Afterwards dom0
> > and guest vCPUs start losing timer interrupts permanently (RCU stalls
> > with a frozen timer-softirq count while dynticks keeps advancing), so
> > VMs hang silently hours or days later.
> I've recently noticed that crashing with interrupts off doesn't reboot
> any more.  I thought it was something local to my development, but
> apparently not if you're seeing the same.
>
> > I've attached the full xl dmesg log of the last crash, and I'm happy
> > to provide more if needed. I can also test patches.
> >
> > Thanks for your work, and thanks in advance for taking a look.
> >
> > Peter
> Thankyou,
>
> ~Andrew
I have full xl dmesg logs for almost all crashes. Happy to send any of
them.

Peter