Re: Linux 6.13-rc3 many different panics in Xen PV dom0

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

On Fri, Dec 20, 2024 at 02:48:52AM +0100, Marek Marczykowski-Górecki wrote:
> On Thu, Dec 19, 2024 at 05:14:52PM +0100, Marek Marczykowski-Górecki wrote:
> > Hi,
> > 
> > It crashes on boot like below, most of the times. But sometimes (rarely)
> > it manages to stay alive. Below I'm pasting few of the crashes that look
> > distinctly different, if you follow the links, you can find more of
> > them. IMHO it looks like some memory corruption bug somewhere. I tested
> > also Linux 6.13-rc2 before, and it had very similar issue.
> > 
> > The traces below are all from nested virt (Xen inside KVM), tests with
> > Xen directly on the hardware are still in progress. But -rc2 failed all
> > of them too, so if it's the same issue, I guess they will looks similar.
> 
> Yes, on real hardware it crashes too.

6.13-rc4 fails the same way.

> I tried to enable KASAN, but that didn't worked out:
> 
> (XEN) d0 has maximum 416 PIRQs
> (XEN) *** Building a PV Dom0 ***
> (XEN)  Xen  kernel: 64-bit, lsb
> (XEN)  Dom0 kernel: 64-bit, lsb, paddr 0x200000 -> 0x7600000
> (XEN) PHYSICAL MEMORY ARRANGEMENT:
> (XEN)  Dom0 alloc.:   0000000260000000->0000000268000000 (1005377 pages to be 
> allocated)
> (XEN)  Init. ramdisk: 000000027d741000->000000027ffff207
> (XEN) VIRTUAL MEMORY ARRANGEMENT:
> (XEN)  Loaded kernel: ffffffff80200000->ffffffff87600000
> (XEN)  Phys-Mach map: 0000008000000000->0000008000800000
> (XEN)  Start info:    ffffffff87600000->ffffffff876004b8
> (XEN)  Page tables:   ffffffff87601000->ffffffff87640000
> (XEN)  Boot stack:    ffffffff87640000->ffffffff87641000
> (XEN)  TOTAL:         ffffffff80000000->ffffffff87800000
> (XEN)  ENTRY ADDRESS: ffffffff8615da50
> (XEN) Dom0 has maximum 2 VCPUs
> (XEN) Initial low memory virq threshold set at 0x4000 pages.
> (XEN) Scrubbing Free RAM in background
> (XEN) Std. Loglevel: All
> (XEN) Guest Loglevel: All
> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
> (XEN) Freed 684kB init memory
> (XEN) d0v0 Unhandled: vec 14, #PF[0002]
> (XEN) Pagetable walk from fffffbfff0900fc6:
> (XEN)  L4[0x1f7] = 0000000000000000 ffffffffffffffff
> (XEN) domain_crash_sync called from entry.S: fault at ffff82d0402ebdec 
> x86_64/entry.S#create_bounce_frame+0x14c/0x170
> (XEN) Domain 0 (vcpu#0) crashed on cpu#0:
> (XEN) ----[ Xen-4.19.0  x86_64  debug=n  Not tainted ]----
> (XEN) CPU:    0
> (XEN) RIP:    e033:[<ffffffff8614ff32>]
> (XEN) RFLAGS: 0000000000000286   EM: 1   CONTEXT: pv guest (d0v0)
> (XEN) rax: ffffffff860d8000   rbx: ffffffff87600000   rcx: 00000000c0000101
> (XEN) rdx: 3be9e05ee5ed7ef7   rsi: ffffffff87600000   rdi: fffffbfff0900fc6
> (XEN) rbp: ffffffff84807f48   rsp: ffffffff84807df0   r8:  0000000000000000
> (XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
> (XEN) r12: dffffc0000000000   r13: 0000000000000000   r14: 0000000000000000
> (XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 0000000000340660
> (XEN) cr3: 0000000267601000   cr2: fffffbfff0900fc6
> (XEN) fsb: 0000000000000000   gsb: ffffffff860d8000   gss: 0000000000000000
> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
> (XEN) Guest stack trace from rsp=ffffffff84807df0:
> (XEN)    00000000c0000101 0000000000000000 0000000000000002 ffffffff8614ff32
> (XEN)    000000010000e030 0000000000010086 ffffffff84807e30 000000000000e02b
> (XEN)    0000000041b58ab3 ffffffff845f8030 ffffffff8614fed0 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    ffffffff8615da6f 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000
> (XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds.
> (XEN) Resetting with ACPI MEMORY or I/O RESET_REG.
> 
> > Who should I CC here? The failures are all over the place... linux-mm?
> 
> -- 
> Best Regards,
> Marek Marczykowski-Górecki
> Invisible Things Lab



-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachment: signature.asc
Description: PGP signature