Re: [PATCH] x86/ucode: Remove MICROCODE_UPDATE_TIMEOUT and associated panic()

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 19/08/2026 12:56 pm, Jan Beulich wrote:
> On 19.08.2026 13:13, Andrew Cooper wrote:
>> On 18/08/2026 3:31 pm, Jan Beulich wrote:
>>> On 18.08.2026 15:07, Andrew Cooper wrote:
>>>> Panicing in the case of a timeout turns out to be about the worst possible
>>>> action Xen can take.  It leaves all other APs waiting on the condition
>>>> variable, some in NMI context.  As a result, they fail to be shot down and
>>>> dump state for kexec crash analysis.
>>> At the same time there likely isn't much to be learned from a kexec dump, as
>>> the source of the issue is in the CPU, not in Xen.
>> The single most valuable print message I've ever added to Xen is the one
>> which reports which CPUs didn't respond to NMIs.  Up until now, it has
>> always highlighted hardware issues.
>>
>> Despite my wish to remove this panic specifically, there is still
>> information to be gained from kexec in a similar scenario.
> That is you think of a system without console, where those log messages would
> only be possible to fish out of the dump. Fair enough.

Yes.  This covers approximately all of XenServer deployments.

>
>>> Further, this code runs with the watchdog disabled. If there's truly no
>>> progress anymore, how would one know from the outside whether the system is
>>> dead altogether vs the control CPU still kicking around?
>> The scenario you describe can only occur if the BSP accepts the
>> microcode successfully, and one of the APs locks up properly.
>>
>> If the BSP locks up, we never get as far as deciding to panic(), and the
>> system hangs already.
>>
>> If we have a bad microcode, it is far more likely for the BSP to hang
>> than for the BSP to work one of the APs hang.
> Hmm, probably. (I'm always having in mind the one old system I have where only
> the primary cores on each socket get ucode updated by firmware, with secondary
> cores needing us to deal with them.) Still somewhat hesitantly:
> Acked-by: Jan Beulich <jbeulich@xxxxxxxx>

Thanks.

>
>>>> Microcode Loading on Granite Rapids takes about 4.5s of wallclock time, 
>>>> far in
>>>> excess of the of the arbitrary 1s Xen allows.  This time is spent in the 
>>>> WRMSR
>>>> to load the blob, and there's nothing the system can do but to sit and 
>>>> wait.
>>>> Despite the delay, the system as a whole does survive.
>>> For this I wonder whether the log message issues after 1 sec is adequate. If
>>> we know things can take this long, wouldn't we better issue the log message
>>> no earlier than, say, 5s * nr_sockets?
>> I have some work there not submitted yet, which would periodically print
>> a message.  That at least gives you slight signs of life.
>>
>> But nothing involving nr_sockets.  It's far more often wrong than it is
>> right, and that's not how our algorithm scales.
>>
>> GNR has gone from milliseconds to 4.5s.  Putting the limit at 5s is just
>> going to need another change in a year or two.
> That's one possibility, yes. The other is that they realize that they have to
> bring processing time back down.

I'm still trying to get details out of Intel, but I strongly suspect the
massive jump here is because of the introduction of the Ucode Staging
Buffer.

Even the architectural description is:

IA32_MCU_ENUMERATION (0x7b), bit 4: MCU_STAGING
"When set to 1, indicates that the microcode update staging capability
is supported by the processor. When supported, the use of the MCU
staging capability is recommended to reduce the latency of the
IA32_BIOS_UPDT_TRIG operation."

which all-but-says "high latency now expected".

When the staging buffer is primed properly, it gets back down to a
reasonable time, but I have no idea when we're going to be able to
support that, and in the meantime running xen-ucode needs to not crash.

One other idea I had:

If we rearrange the loop to have a periodic "taken $X seconds" (shows
liveness), and maybe at 40s decide to panic (by passing through the out:
label first so we release the APs, so they can crash more cleanly), then
that might be acceptable.

By 40s, the system isn't surviving even if the cores do all come back.

~Andrew