Re: [BUG] Xen BUG at mce_amd.c:100 (mc_ec2type) on Zen 3; host left wedged, no reboot

[Jan Beulich <jbeulich@xxxxxxxx>]

On 10.11.2026 10:59, Peter Deggendorfer wrote:
> Am 02.11.2026 um 13:54 schrieb Jan Beulich:
>> On 31.10.2026 14:44, Peter Deggendorfer wrote:
>>> As requested, I added a printk to mcheck_mca_logout(), and the BUG() in
>>> mc_ec2type() is still replaced. These are the only changes against the
>>> Debian package:
>>>
>>> --- a/xen/arch/x86/cpu/mcheck/mce_amd.c
>>> +++ b/xen/arch/x86/cpu/mcheck/mce_amd.c
>>> @@ -97,7 +97,7 @@
>>>        if ( errorcode & MC_EC_TLB_TYPE )
>>>            return MC_EC_TLB_TYPE;
>>>        /* Unreached */
>>> -    BUG();
>>> +    printk(XENLOG_WARNING "MCE: unknown AMD error code %#x\n", errorcode);
>>>        return 0;
>>>    }
>>>
>>> --- a/xen/arch/x86/cpu/mcheck/mce.c
>>> +++ b/xen/arch/x86/cpu/mcheck/mce.c
>>> @@ -304,3 +304,6 @@
>>>            if ( !(status & MCi_STATUS_VAL) )
>>>                continue; /* this bank has no valid telemetry */
>>>
>>> +        if ( who == MCA_POLLER )
>>> +            printk(XENLOG_WARNING "MCE: CPU%u bank %d status %#"PRIx64"\n",
>>> +                   smp_processor_id(), i, status);
>>>
>>>
>>> We now have some data points:
>>>
>>> (XEN) [2026-10-27 13:42:10.588] MCE: CPU2 bank 23 status 0x830824448b48ffff
>>> (XEN) [2026-10-27 18:19:55.593] MCE: CPU2 bank 23 status 0xffffffffffffffff
>>> (XEN) [2026-10-27 22:11:10.596] MCE: CPU2 bank 17 status 0xffff832036a40048
>>> (XEN) [2026-10-27 22:11:10.596] MCE: unknown AMD error code 0x48
>>> (XEN) [2026-10-28 00:33:55.597] MCE: CPU2 bank 10 status 0xffff82d04022a8af
>>> (XEN) [2026-10-28 05:26:40.599] MCE: CPU2 bank 17 status 0xffff82d0402408e3
>>> (XEN) [2026-10-29 23:17:25.618] MCE: CPU2 bank 17 status 0xffff82d04022a8af
>>> (XEN) [2026-10-30 03:25:10.622] MCE: CPU2 bank 23 status 0xffffffffffffffff
>>> (XEN) [2026-10-30 10:05:55.625] MCE: CPU2 bank 10 status 0xffffffffffffffff
>>> (XEN) [2026-10-31 00:58:10.625] MCE: CPU2 bank 17 status 0xffff888100b872e8
>>> (XEN) [2026-10-31 00:58:10.625] MCE: unknown AMD error code 0x72e8
>>> (XEN) [2026-10-31 04:17:55.632] MCE: CPU2 bank 23 status 0xffff831f00006000
>>> (XEN) [2026-10-31 04:17:55.632] MCE: unknown AMD error code 0x6000
>>>
>>> Only 3 of these would have hit the BUG(); the others have UC clear or
>>> an error code that happens to contain a type bit.
>>>
>>> The values look like pointers rather than status registers:
>>>
>>>     ffff82d040......  Xen text/data -- the prefix of every RIP in my
>>>                       earlier panic dumps
>>>     ffff831f........ / ffff8320........  same range as rsp in those
>>>                       dumps (e.g. ffff831fe6b3fc38, ffff832036a87c38)
>>>     ffff888100b872e8  dom0 Linux range?
>> Plus the very first of the entries above is a pointer value rotated by 16 
>> bits.
>>
>>> Two things that make me doubt these are real status values at all:
>>> 0xffff82d04022a8af appears twice, 47 hours apart, for two different
>>> banks (10 and 17). And all timestamps are congruent mod 15 seconds,
>>> i.e. this is the poller hitting it on a small fraction of passes.
>> There's no question that the values are garbage. We just need to find out why
>> that is. As we have to explore all (remote) possibilities, first of all - you
>> don't play any games involving the XEN_MC_msrinject hypercall, do you? For
>> the immediate purpose here, and since ...
> 
> No, I never used XEN_MC_msrinject hypercall on this host; it's a
> plain Debian trixie installation.
> 
> 
>>> I don't know where the read goes wrong.
>> ... I can't make sense of the behavior either, I'm inclined to suggest that
>> you add this hunk to your debugging patch:
>>
>> --- a/xen/arch/x86/cpu/mcheck/mce.h
>> +++ b/xen/arch/x86/cpu/mcheck/mce.h
>> @@ -79,7 +79,6 @@
>>   static inline uint64_t mca_rdmsr(unsigned int msr)
>>   {
>>       uint64_t val;
>> -    if (intpose_lookup(smp_processor_id(), msr, &val) == NULL)
>>           rdmsrl(msr, val);
>>       return val;
>>   }

Can you please replace this by

--- a/xen/arch/x86/cpu/mcheck/mce.h
+++ b/xen/arch/x86/cpu/mcheck/mce.h
@@ -79,8 +79,18 @@ extern bool intpose_inval(unsigned int c
 static inline uint64_t mca_rdmsr(unsigned int msr)
 {
     uint64_t val;
-    if (intpose_lookup(smp_processor_id(), msr, &val) == NULL)
-        rdmsrl(msr, val);
+
+    if ( !intpose_lookup(smp_processor_id(), msr, &val) )
+    {
+        unsigned long lo = 0x555555555555UL, hi = 0xaaaaaaaaU;
+
+        asm volatile ( "rdmsr"
+                       : "+a" (lo), "+d" (hi)
+                       : "c" (msr) );
+
+        val = (hi << 32) | lo;
+    }
+
     return val;
 }
 
>From ...

>>> /dev/mcelog records match the log lines one-to-one (same banks,
>>> same order, all CPU 2), but carry status zero in every case. The record's
>>> status comes from the second read in mca_init_bank(), before
>>> mcabank_clear(), so that second read returned 0.
>> Based on this observation, another thing you may want to try is extend the
>> existing hunk you have to
>>
>> --- a/xen/arch/x86/cpu/mcheck/mce.c
>> +++ b/xen/arch/x86/cpu/mcheck/mce.c
>> @@ -304,6 +304,17 @@
>>            if ( !(status & MCi_STATUS_VAL) )
>>                continue; /* this bank has no valid telemetry */
>>
>> +        if ( who == MCA_POLLER )
>> +        {
>> +            printk("MCE: CPU%u bank %d status %#"PRIx64"\n",
>> +                   smp_processor_id(), i, status);
>> +            status = mca_rdmsr(MSR_IA32_MCx_STATUS(i));
>> +            printk("MCE: CPU%u bank %d -> %#"PRIx64"\n",
>> +                   smp_processor_id(), i, status);
>> +            if ( !(status & MCi_STATUS_VAL) )
>> +                continue;
>> +        }
>> +
>>           /*
>>            * For Intel Latest CPU CMCI/MCE Handler caller, we need to
>>            * decide whether to clear bank by MCi_STATUS bit value such as
>>
> 
> 
> Since 2026-11-03 the host runs Xen 4.20.5-pre (Debian
> 4.20.4+39-gf1e1321c87-0+deb13u1) with your two debugging hunks
> (intpose_lookup() removed from mca_rdmsr(), re-read in the poller) and
> your "x86/MCE: reduce redundant code" series (both patches). In 7 days:
> 
> 
> (XEN) [2026-11-03 13:54:50.378] MCE: CPU2 bank 25 status 0x8000001fe67e1163
> (XEN) [2026-11-03 13:54:50.378] MCE: CPU2 bank 25 -> 0
> (XEN) [2026-11-03 14:52:20.360] MCE: CPU2 bank 7 status 0x8000000080000000
> (XEN) [2026-11-03 14:52:20.360] MCE: CPU2 bank 7 -> 0
> (XEN) [2026-11-03 20:01:20.264] MCE: CPU2 bank 25 status 0xffff82d04022a8ef
> (XEN) [2026-11-03 20:01:20.264] MCE: CPU2 bank 25 -> 0
> (XEN) [2026-11-05 14:31:19.276] MCE: CPU2 bank 25 status 0x8000001fe67e1163
> (XEN) [2026-11-05 14:31:19.276] MCE: CPU2 bank 25 -> 0
> (XEN) [2026-11-06 11:27:18.812] MCE: CPU2 bank 24 status 0xffff832036a754c0
> (XEN) [2026-11-06 11:27:18.812] MCE: CPU2 bank 24 -> 0
> (XEN) [2026-11-06 18:26:48.671] MCE: CPU2 bank 7 status 0xffff831fe6b154c0
> (XEN) [2026-11-06 18:26:48.671] MCE: CPU2 bank 7 -> 0
> (XEN) [2026-11-07 06:31:48.345] MCE: CPU2 bank 16 status 0xffff8313f11d2010
> (XEN) [2026-11-07 06:31:48.345] MCE: CPU2 bank 16 -> 0
> (XEN) [2026-11-07 09:41:48.243] MCE: CPU2 bank 16 status 0xffffffffffffffff
> (XEN) [2026-11-07 09:41:48.243] MCE: CPU2 bank 16 -> 0
> (XEN) [2026-11-07 21:32:17.875] MCE: CPU2 bank 7 status 0xffff83203a5ea2d0
> (XEN) [2026-11-07 21:32:17.875] MCE: CPU2 bank 7 -> 0
> (XEN) [2026-11-08 06:25:47.633] MCE: CPU2 bank 16 status 0xffffffffffffffff
> (XEN) [2026-11-08 06:25:47.633] MCE: CPU2 bank 16 -> 0
> (XEN) [2026-11-08 15:11:17.390] MCE: CPU2 bank 16 status 0xffff82d040588f80
> (XEN) [2026-11-08 15:11:17.390] MCE: CPU2 bank 16 -> 0
> (XEN) [2026-11-09 07:29:46.980] MCE: CPU2 bank 16 status 0xffffffffffffffff
> (XEN) [2026-11-09 07:29:46.980] MCE: CPU2 bank 16 -> 0
> (XEN) [2026-11-09 10:49:46.880] MCE: CPU2 bank 7 status 0xffffffffffffffff
> (XEN) [2026-11-09 10:49:46.880] MCE: CPU2 bank 7 -> 0
> (XEN) [2026-11-09 13:09:16.822] MCE: CPU2 bank 7 status 0xffffffffffffffff
> (XEN) [2026-11-09 13:09:16.822] MCE: CPU2 bank 7 -> 0

... this it almost looks as if the first RDMSR is "skipped" altogether. The
changed debugging hunk might further confirm that, at which point I think
we need to assume that there is some issue with the hardware here (which is
already suggested by all instances being confined to CPU2).

Jan