Re: Linux 6.13-rc3 many different panics in Xen PV dom0

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 02/01/2025 6:54 pm, Marek Marczykowski-Górecki wrote:
> On Thu, Jan 02, 2025 at 01:24:21PM +0100, Marek Marczykowski-Górecki wrote:
>> On Thu, Jan 02, 2025 at 12:30:10PM +0100, Juergen Gross wrote:
>>> On 02.01.25 11:20, Jürgen Groß wrote:
>>>> On 19.12.24 17:14, Marek Marczykowski-Górecki wrote:
>>>>> Hi,
>>>>>
>>>>> It crashes on boot like below, most of the times. But sometimes (rarely)
>>>>> it manages to stay alive. Below I'm pasting few of the crashes that look
>>>>> distinctly different, if you follow the links, you can find more of
>>>>> them. IMHO it looks like some memory corruption bug somewhere. I tested
>>>>> also Linux 6.13-rc2 before, and it had very similar issue.
>>>> ...
>>>>
>>>>> Full log:
>>>>> https://openqa.qubes-os.org/tests/122879/logfile?filename=serial0.txt
>>>> I can reproduce a crash with 6.13-rc5 PV dom0.
>>>>
>>>> What is really interesting in the logs: most crashes seem to happen right
>>>> after a module being loaded (in my reproducer it was right after loading
>>>> the first module).
>>>>
>>>> I need to go through the 6.13 commits, but I think I remember having seen
>>>> a patch optimizing module loading by using large pages for addressing the
>>>> loaded modules. Maybe the case of no large pages being available isn't
>>>> handled properly.
>>> Seems I was right.
>>>
>>> For me the following diff fixes the issue. Marek, can you please confirm
>>> it fixes your crashes, too?
>> Thanks for looking into it!
>> Will do, I've pushed it to
>> https://github.com/QubesOS/qubes-linux-kernel/pull/662, CI will build it
>> and then I'll post it to openQA.
> It is much better!
>
> Tests are still running, but I already see that many are green. There is
> one issue (likely unrelated to this change) - sys-usb (HVM domU with USB
> controllers passed through) crashes on a system with Raptor Lake CPU
> (only, others, including ADL and MTL look fine):
>
> [   75.770849] Bluetooth: Core ver 2.22
> [   75.770866] Oops: general protection fault, probably for non-canonical 
> address 0xc9d2315bc82c3bbd: 0000 [#1] PREEMPT SMP NOPTI
> [   75.770880] CPU: 0 UID: 0 PID: 923 Comm: (udev-worker) Not tainted 
> 6.13.0-0.rc5.2.qubes.1.fc41.x86_64 #1
> [   75.770890] Hardware name: Xen HVM domU, BIOS 4.19.0 01/02/2025
> [   75.770897] RIP: 0010:msft_monitor_device_del+0x93/0x170 [bluetooth]
> [   75.770924] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0 65 21 <26> 2b 
> 8b ad 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

es sub 0x3ad(%rbx),%ecx

I highly doubt that's an instruction that the compiler really put out
for this function.

The preceding bytes are "shlb 0x21(%rbp)" which isn't completely
implausible, but the surrounding 0's very much are.

This looks very fishy, and either looks like DMA hitting .text, or
module handling getting it's regions wrong.

~Andrew