Re: Linux 6.13-rc3 many different panics in Xen PV dom0

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

On Thu, Dec 19, 2024 at 05:14:52PM +0100, Marek Marczykowski-Górecki wrote:
> Hi,
> 
> It crashes on boot like below, most of the times. But sometimes (rarely)
> it manages to stay alive. Below I'm pasting few of the crashes that look
> distinctly different, if you follow the links, you can find more of
> them. IMHO it looks like some memory corruption bug somewhere. I tested
> also Linux 6.13-rc2 before, and it had very similar issue.
> 
> The traces below are all from nested virt (Xen inside KVM), tests with
> Xen directly on the hardware are still in progress. But -rc2 failed all
> of them too, so if it's the same issue, I guess they will looks similar.

Yes, on real hardware it crashes too.

I tried to enable KASAN, but that didn't worked out:

(XEN) d0 has maximum 416 PIRQs
(XEN) *** Building a PV Dom0 ***
(XEN)  Xen  kernel: 64-bit, lsb
(XEN)  Dom0 kernel: 64-bit, lsb, paddr 0x200000 -> 0x7600000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000260000000->0000000268000000 (1005377 pages to be 
allocated)
(XEN)  Init. ramdisk: 000000027d741000->000000027ffff207
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff80200000->ffffffff87600000
(XEN)  Phys-Mach map: 0000008000000000->0000008000800000
(XEN)  Start info:    ffffffff87600000->ffffffff876004b8
(XEN)  Page tables:   ffffffff87601000->ffffffff87640000
(XEN)  Boot stack:    ffffffff87640000->ffffffff87641000
(XEN)  TOTAL:         ffffffff80000000->ffffffff87800000
(XEN)  ENTRY ADDRESS: ffffffff8615da50
(XEN) Dom0 has maximum 2 VCPUs
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 684kB init memory
(XEN) d0v0 Unhandled: vec 14, #PF[0002]
(XEN) Pagetable walk from fffffbfff0900fc6:
(XEN)  L4[0x1f7] = 0000000000000000 ffffffffffffffff
(XEN) domain_crash_sync called from entry.S: fault at ffff82d0402ebdec 
x86_64/entry.S#create_bounce_frame+0x14c/0x170
(XEN) Domain 0 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-4.19.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e033:[<ffffffff8614ff32>]
(XEN) RFLAGS: 0000000000000286   EM: 1   CONTEXT: pv guest (d0v0)
(XEN) rax: ffffffff860d8000   rbx: ffffffff87600000   rcx: 00000000c0000101
(XEN) rdx: 3be9e05ee5ed7ef7   rsi: ffffffff87600000   rdi: fffffbfff0900fc6
(XEN) rbp: ffffffff84807f48   rsp: ffffffff84807df0   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: dffffc0000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 0000000000340660
(XEN) cr3: 0000000267601000   cr2: fffffbfff0900fc6
(XEN) fsb: 0000000000000000   gsb: ffffffff860d8000   gss: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff84807df0:
(XEN)    00000000c0000101 0000000000000000 0000000000000002 ffffffff8614ff32
(XEN)    000000010000e030 0000000000010086 ffffffff84807e30 000000000000e02b
(XEN)    0000000041b58ab3 ffffffff845f8030 ffffffff8614fed0 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    ffffffff8615da6f 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000
(XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds.
(XEN) Resetting with ACPI MEMORY or I/O RESET_REG.

> Who should I CC here? The failures are all over the place... linux-mm?

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachment: signature.asc
Description: PGP signature