Re: [BUG] Xen BUG at mce_amd.c:100 (mc_ec2type) on Zen 3; host left wedged, no reboot

[Jan Beulich <jbeulich@xxxxxxxx>]

On 25.10.2026 13:32, Peter Deggendorfer wrote:
> 
>> Hi Andrew,
>>
>> Thanks a lot for the quick response.
>>
>>
>>> On 23/10/2026 8:32 am, Peter Deggendorfer wrote:
>>>> Hey Xen Team,
>>>>
>>>> I'm hitting a reproducible hypervisor panic in the AMD non-fatal MCE
>>>> polling path on a Ryzen 9 5950X (Zen 3). Eight occurrences since
>>>> 2026-09-09, between 12 hours and 2 weeks apart, always CPU 2.
>>>>
>>>> Environment
>>>> -----------
>>>> Xen:   4.20.4-pre (Debian 4.20.3+127-gc42374a105-0+deb13u1), and still
>>>>         present in 4.20.5-pre (Debian 4.20.4+39-gf1e1321c87-0+deb13u1)
>>>> dom0:  Linux 6.12.111+deb13-amd64 (Debian trixie), PV, 32 vCPUs
>>>> CPU:   AMD Family 25 (0x19) Model 33 (0x21) Stepping 2, Ryzen 9 5950X,
>>>>         microcode 0x0a201210
>>>> Cmdline: dom0_mem=4096M,max:4096M ucode=scan console_timestamps=datems
>>>>
>>>> The panic
>>>> ---------
>>>> (XEN) [2026-10-19 08:58:18.727] Xen BUG at
>>>> arch/x86/cpu/mcheck/mce_amd.c:100
>>>> (XEN) CPU:    2
>>>> (XEN) RIP:    e008:[<ffff82d04028a650>]
>>>> mce_amd.c#mc_ec2type.part.0+0/0x10
>>>> (XEN) rbx: 0000000000000445   r12: 0000000000000011   r15:
>>>> 0000000000000446
>>>> (XEN) Xen code around <ffff82d04028a650>:
>>>> (XEN)  90 90 90 90 90 90 90 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90
>>>> 90 90
>>>> (XEN) Xen call trace:
>>>> (XEN)    [<ffff82d04028a650>] R mce_amd.c#mc_ec2type.part.0+0/0x10
>>>> (XEN)    [<ffff82d04028a699>] S mc_amd_recoverable_scan+0x39/0x40
>>>> (XEN)    [<ffff82d04028cfdd>] S mcheck_mca_logout+0x42d/0x840
>>>> (XEN)    [<ffff82d04028a35c>] S
>>>> amd_nonfatal.c#mce_amd_checkregs+0x1c/0x100
>>>> (XEN)    [<ffff82d040240e8c>] S smp_call_function_interrupt+0x4c/0x90
>>>> (XEN)    [<ffff82d0403123d5>] S do_IRQ+0x2a5/0x650
>>>> (XEN)    [<ffff82d040201915>] S common_interrupt+0x135/0x150
>>>> (XEN)    [<ffff82d0402883ca>] S 
>>>> cpu_idle.c#acpi_idle_do_entry+0x8a/0x140
>>>> (XEN)    [<ffff82d0402887cf>] S
>>>> cpu_idle.c#acpi_processor_idle+0x1ef/0x650
>>>> (XEN)    [<ffff82d040302cf1>] S domain.c#idle_loop+0xc1/0x130
>>>> (XEN) Panic on CPU 2
>>>>
>>>> The outlined mc_ec2type.part.0 contains nothing but the ud2, so this is
>>>> the "unreachable" default branch of mc_ec2type() -- the extended error
>>>> code matches none of MC_EC_TLB_TYPE, MC_EC_MEM_TYPE, MC_EC_BUS_TYPE.
>>> I think it's safe to just drop the BUG().
>>>
>>> That code is literally worse than useless, because if nothing else it's
>>> just hiding the diagnostics.
>>>
>>> I guess it means AMD have added new statuses.
>>>
>>>> Across the eight occurrences, r12 appears to be the bank index from the
>>>> loop in mcheck_mca_logout(), with rbx == 0x401 + 4 * r12 (MCi_STATUS)
>>>> and r15 == rbx + 1. Three different banks are involved: 10, 17 and 23.
>>>>
>>>> Not a hardware defect
>>>> ---------------------
>>>> - Started within days of upgrading two hosts from Debian bookworm
>>>>    (Xen 4.17) to trixie (Xen 4.20). They had run 3 years on 4.17 
>>>> with no
>>>>    incident.
>>>> - Three different banks, so not a single failing block.
>>>> - Updating to the current Debian package (4.20.5-pre, built 2026-10-16)
>>>>    changed nothing.
>>>>
>>>> Possibly related: 532988dccc19 ("x86/mcheck: fix mca_allbanks
>>>> updating").
>>>>
>>>> Also possibly related on the Linux side: 0422b07bc4c2 ("x86/mce/amd:
>>>> Filter bogus hardware errors on Zen3 clients"), where bogus MCA values
>>>> on Zen 3 are filtered rather than treated as errors.
>>>> https://ratatoskr.run/stable/2026/04/7143905
>>>  From above, you're on a Vermeer, not a Cezanne, so I don't think that
>>> Linux patch is relevant.
>>>
>>> However, your observation about "fix mca_allbanks updating" quite
>>> possibly is.  That was backported into Xen 4.20.1 as 247b708b1400a, and
>>> Debian has 4.20.3
>>>
>>> Are you able to experiment with that patch reverted?
>>>
>>> Alternatively, are you able to experiment with the BUG() removed?
>>
>> I went with dropping the BUG() rather than the revert.
>> Reverting 247b708b1400a should stop any bank from being read at
>> all, which is effectively what mce=false already does here, so I don't
>> expect it to tell us anything new. I'll run it too if you'd like.
>>
>> I'm not familiar with the Xen codebase, but I've built a package with
>> mc_ec2type() changed to:
>>
>>     if ( errorcode & MC_EC_TLB_TYPE )
>>         return MC_EC_TLB_TYPE;
>>
>>     printk(XENLOG_WARNING "MCE: unknown AMD error code %#x\n", 
>> errorcode);
>>
>>     return 0;
>>
>>
>> Crashes have been 12 hours to 2 weeks apart, so it may take a while
>> before I can report back.
> 
> After ~31 hours of uptime:
> 
> (XEN) [2026-10-25 04:06:27.260] MCE: unknown AMD error code 0
> 
> No panic, no wedged CPU, no RCU stalls, no hung guests. Host and all
> VMs running normally.
> 
> 
> I also dumped /dev/mcelog in dom0 and got the following records:
> 
> 00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
> 00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
> 00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
> 00000030: 0000 0000 0000 0000 0200 0000 0000 0000  ................
> 00000040: 0010 0201 0200 0000 0000 0000 0200 0000  ................
> 00000050: 1c01 0000 0000 0000 0000 0000 0000 0000  ................
> 00000060: 0000 0000 0000 0000 0000 0000 0000 0000  ................
> 00000070: 0000 0000 0000 0000 0000 0000 0000 0000  ................
> 00000080: 0000 0000 0000 0000 0000 0000 0000 0000  ................
> 00000090: 0000 0000 0000 0000 0000 0000 0000 0000  ................
> 000000a0: 0000 0000 0000 0000 0200 0000 0000 0000  ................
> 000000b0: 0017 0201 0200 0000 0000 0000 0200 0000  ................
> 000000c0: 1c01 0000 0000 0000 0000 0000 0000 0000  ................
> 000000d0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
> 
> 
>>
>>
>>>> Diagnostics / workaround
>>>> ------------------------
>>>> mce_verbosity=verbose yields nothing useful.
>>>> max_cstate=1 made no difference. mce=false is the only thing that stops
>>>> it, and it is what I'm running now.
>>> The fact that mce=false isn't causing machine resets means it's clearly
>>> something the hardware thinks isn't fatal.
>>>
>>> But, I am concerned by the fact it's always on CPU 2, and multiple
>>> banks.  Is this true for both systems?
>>
>> To clarify one thing from my report: it is actually a single host whose
>> complete hardware (board, CPU, RAM) was replaced on
>> 2026-10-07, keeping only the disks and thus the installation. The
>> crashes happened both before and after that swap. 10 more hosts
>> with same hardware still run on bookworm (xen 4.17) for years now
>> without any incidents.
>>
>> So yes, always CPU 2. The bank index varies: on the current hardware
>> I have seen banks 23, 17 and 10 across four crashes, and banks 23 and
>> 17 on the previous hardware.
> 
> Regarding the output of /dev/mcelog: if my decoding is right, three
> things stand out. MCi_STATUS is zero, i.e. these banks are reporting
> nothing at all -- no error, no garbage value.

Well, things can't quite be that way: If MCi_STATUS was zero, the first
check in mc_amd_recoverable_scan() would prevent mc_ec2type() from
being reached in the first place. Its caller also checks MCi_STATUS_VAL
to be set.

What I find puzzling is this:

    errorcode = status & (MCi_STATUS_MCA | MCi_STATUS_MSEC);
    ectype = mc_ec2type(errorcode);

With errorcode (and mc_ec2type()'s parameter type) being uint16_t, what
use is it to include MCi_STATUS_MSEC in the mask? (Yet that's likely
unrelated to the issue at hand.) Considering how mc_ec2type() works, I
see no reason why status couldn't be passed to it directly (with the
parameter type widened to uint32_t). Then the BUG() there could perhaps
become BUG_ON(errorcode).

> Both records are again
> from CPU 2, and bank 23 is one of the banks that appeared in the
> panics, bank 16 is new.

Isn't it possible that the lack of 532988dccc19 ("x86/mcheck: fix
mca_allbanks updating") simply hid the issue previously?

Jan