[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: domU crash when resuming from suspend - Xen 4.22


  • To: Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>
  • From: Frediano Ziglio <freddy77@xxxxxxxxx>
  • Date: Sun, 19 Jul 2026 08:31:28 +0100
  • Arc-authentication-results: i=1; mx.google.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=yXhVGLrq8BbOV499aq5difJP2J7h0KknahtbRk8BpgU=; fh=xICrRNGL1/wiAEX7CoGX/RRRWm4TcI+zcC/u/Ri/Jas=; b=lbAlrqQSgtOGOSETIvjQJy77C3/sT9FlIi4K0w8M37AbCzIgc57cNTNo4AaJyKq8zz Knw7KojqG0cj2gZ3Bi72QRWGtwPI2pHvHgsz7pX8P4wm8BjEVGYXjLT1AUrEoslHym0E jLTT6uMjyvJkfBwISHvZ4SFhJcm2Yw0dh4mqvB0bqovXqvFMMHwvnG3Un0hGlhY4lzbf pehJ4n41tUbApQJ7lMVvTH37ATqesJIqG9NUQlNZT8femeQvtxJ5p7zGCSKUCMNln+LI 7weGkZ4vabSMnP7qJdIpGMfyOHbldBeQWkfVCtWvePzoHED2kDLDiw+4ifcJLfCshQEa bgcw==; darn=lists.xenproject.org
  • Arc-seal: i=1; a=rsa-sha256; t=1784446303; cv=none; d=google.com; s=arc-20260327; b=VHMBT4zqUfv9QOc5VgY1tFtlPBXuoClDvlzSBBTrwj+1GcBmECKdFjp7/GuzwaclrG wJzRrN8W2iPEu05jrlKmxMMPtOJgzeKfWJ5A3mDFVkEl3uazlhJKZM98dJey4WDbRhIr P9/k7KHwZjjXQiU7QqVyyfoRM8kCmBWq3WquNNLLHsfXrn8DYiBewaWQPLYwWc7GBzWQ yJX0PI5gT8RxGimuxCriUjk3rciFKNeL/qcaHiyrUG5oFfrZZJftZwJIJwi6v3dCBb2+ T5yDxMu5/y2s+FLCECiT7uEEHWSR43eyrhNktjref9K3nVQmMpvP6Bg3vCDNupioyOGf BZ+Q==
  • Authentication-results: eu.smtp.expurgate.cloud; dkim=pass header.s=20251104 header.d=gmail.com header.i="@gmail.com" header.h="Content-Transfer-Encoding:Content-Type:Cc:To:Subject:Message-ID:Date:From:In-Reply-To:References:MIME-Version"
  • Cc: xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>, Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
  • Delivery-date: Sun, 19 Jul 2026 07:32:02 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On Sun, 19 Jul 2026 at 03:00, Marek Marczykowski-Górecki
<marmarek@xxxxxxxxxxxxxxxxxxxxxx> wrote:
>
> On Sat, Jul 18, 2026 at 04:03:29PM +0200, Marek Marczykowski-Górecki wrote:
> > On Fri, Jul 17, 2026 at 09:22:47PM +0100, Frediano Ziglio wrote:
> > > On Tue, 14 Jul 2026 at 18:12, Marek Marczykowski-Górecki
> > > <marmarek@xxxxxxxxxxxxxxxxxxxxxx> wrote:
> > > >
> > > > On Fri, Jun 19, 2026 at 01:24:09PM +0200, Marek Marczykowski-Górecki 
> > > > wrote:
> > > > > Hi,
> > > > >
> > > > > I'm continuing testing 4.22 and found this:
> > > > >
> > > > > When a domU is suspended (even without host suspend) it crashes on
> > > > > resume with:
> > > > >
> > > > >     [   24.374629] BUG: kernel NULL pointer dereference, address: 
> > > > > 0000000000000020
> > > > >     [   24.374647] #PF: supervisor read access in kernel mode
> > > > >     [   24.374657] #PF: error_code(0x0000) - not-present page
> > > > >     [   24.374668] PGD 0 P4D 0
> > > > >     [   24.374675] Oops: Oops: 0000 [#1] SMP NOPTI
> > > > >     [   24.374685] CPU: 0 UID: 0 PID: 19 Comm: migration/0 Not 
> > > > > tainted 6.18.31-1.qubes.13.fc41.x86_64 #1 PREEMPT(full)
> > > > >     [   24.374705] Stopper: multi_cpu_stop+0x0/0x140 <- 
> > > > > __stop_cpus.constprop.0+0x6f/0xb0
> > > > >     [   24.374723] RIP: 0010:multi_cpu_stop+0x63/0x140
> > > > >     [   24.374734] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 
> > > > > 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 
> > > > > ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 
> > > > > 20 41 83 fd
> > > > >     [   24.374764] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
> > > > >     [   24.374775] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 
> > > > > 0000000000000404
> > > > >     [   24.374790] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 
> > > > > 0000000000770ef0
> > > > >     [   24.374801] RBP: 0000000000000002 R08: ffffc900000abc80 R09: 
> > > > > ffffc900000abc8c
> > > > >     [   24.374813] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: 
> > > > > ffffffff81835ac0
> > > > >     [   24.374825] R13: 0000000000000000 R14: 0000000000000000 R15: 
> > > > > 0000000000000000
> > > > >     [   24.374837] FS:  0000000000000000(0000) 
> > > > > GS:ffff88809b369000(0000) knlGS:0000000000000000
> > > > >     [   24.374850] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > >     [   24.374860] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 
> > > > > 0000000000770ef0
> > > > >     [   24.374873] PKRU: 00000000
> > > > >     [   24.374877] Call Trace:
> > > > >     [   24.374884]  <TASK>
> > > > >     [   24.374891]  ? __pfx_multi_cpu_stop+0x10/0x10
> > > > >     [   24.374900]  cpu_stopper_thread+0xa3/0x170
> > > > >     [   24.374908]  ? __pfx_smpboot_thread_fn+0x10/0x10
> > > > >     [   24.374917]  smpboot_thread_fn+0xf3/0x220
> > > > >     [   24.374925]  kthread+0xfc/0x240
> > > > >     [   24.374933]  ? __pfx_kthread+0x10/0x10
> > > > >     [   24.374940]  ? __pfx_kthread+0x10/0x10
> > > > >     [   24.374949]  ret_from_fork+0x158/0x170
> > > > >     [   24.374957]  ? __pfx_kthread+0x10/0x10
> > > > >     [   24.374965]  ret_from_fork_asm+0x1a/0x30
> > > > >     [   24.374973]  </TASK>
> > > > >     [   24.374978] Modules linked in: snd_seq_dummy snd_hrtimer 
> > > > > snd_seq snd_seq_device snd_timer snd soundcore cfg80211 rfkill vfat 
> > > > > fat xenfs binfmt_misc nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 
> > > > > nft_reject nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 
> > > > > nf_tables intel_rapl_msr intel_rapl_common 
> > > > > intel_uncore_frequency_common intel_pmc_ssram_telemetry intel_vsec 
> > > > > polyval_clmulni ghash_clmulni_intel xen_netfront xen_privcmd 
> > > > > xen_gntdev xen_gntalloc xen_blkback xen_evtchn fuse loop nfnetlink 
> > > > > ip_tables overlay xen_blkfront
> > > > >     [   24.375064] CR2: 0000000000000020
> > > > >     [   24.375071] ---[ end trace 0000000000000000 ]---
> > > > >     [   24.375080] RIP: 0010:multi_cpu_stop+0x63/0x140
> > > > >     [   24.375089] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 
> > > > > 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 
> > > > > ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 
> > > > > 20 41 83 fd
> > > > >     [   24.375115] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
> > > > >     [   24.375124] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 
> > > > > 0000000000000404
> > > > >     [   24.375142] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 
> > > > > 0000000000770ef0
> > > > >     [   24.375155] RBP: 0000000000000002 R08: ffffc900000abc80 R09: 
> > > > > ffffc900000abc8c
> > > > >     [   24.375170] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: 
> > > > > ffffffff81835ac0
> > > > >     [   24.375184] R13: 0000000000000000 R14: 0000000000000000 R15: 
> > > > > 0000000000000000
> > > > >     [   24.375200] FS:  0000000000000000(0000) 
> > > > > GS:ffff88809b369000(0000) knlGS:0000000000000000
> > > > >     [   24.375214] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > >     [   24.375226] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 
> > > > > 0000000000770ef0
> > > > >     [   24.375241] PKRU: 00000000
> > > > >     [   24.375247] Kernel panic - not syncing: Fatal exception
> > > > >     [   25.417403] Shutting down cpus with NMI
> > > > >     [   25.417447] Kernel Offset: disabled
> > > > >
> > > > > There is no change on the Linux kernel side (neither dom0 nor domU), 
> > > > > the
> > > > > only change is updating Xen 4.19.4 -> 4.22-rc2 (seems to be already
> > > > > broken on -rc1 too). This happens for both HVM and PVH domains. 
> > > > > Haven't
> > > > > tried PV domU (but dom0 survives host suspend).
> > > >
> > > > Any ideas?
> > > >
> > >
> > > Hi,
> > >    not many ideas but while testing my libxenguest changes I found
> > > similar failures using xl create/save/restore. But I was testing more
> > > PV guests.
> > > Does 4.21 work ?
> >
> > It seems 4.21 also has this issue:
> > https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/15414211523
> > But 4.20 is okay:
> > https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/15414227095
> >
> > I guess I can bisect it further.
>
> So, bisect says the first bad commit is:
>
>     commit 9f892f84c27970840d255aef7e2851b762967de7
>     Author: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
>     Date:   Mon Dec 30 11:49:14 2024 +0000
>
>         x86/domctl: Stop using XLAT_cpu_user_regs()
>
> My test isn't 100% reliable, but I retested parent of this commit a
> couple of times and it was good every time.
>
> I wanted to test reverting it on top of 4.21, but there is a lot of
> FRED-related work on top that makes such revert quite hard, to say the
> least...
>

The generated XLAT_cpu_user_regs was

#define XLAT_cpu_user_regs(_d_, _s_) do { \
    (_d_)->__DECL_REG_LO8(b) = (_s_)->__DECL_REG_LO8(b); \
    (_d_)->__DECL_REG_LO8(c) = (_s_)->__DECL_REG_LO8(c); \
    (_d_)->__DECL_REG_LO8(d) = (_s_)->__DECL_REG_LO8(d); \
    (_d_)->__DECL_REG_LO16(si) = (_s_)->__DECL_REG_LO16(si); \
    (_d_)->__DECL_REG_LO16(di) = (_s_)->__DECL_REG_LO16(di); \
    (_d_)->__DECL_REG_LO16(bp) = (_s_)->__DECL_REG_LO16(bp); \
    (_d_)->__DECL_REG_LO8(a) = (_s_)->__DECL_REG_LO8(a); \
    (_d_)->error_code = (_s_)->error_code; \
    (_d_)->entry_vector = (_s_)->entry_vector; \
    (_d_)->__DECL_REG_LO16(ip) = (_s_)->__DECL_REG_LO16(ip); \
    (_d_)->cs = (_s_)->cs; \
    (_d_)->saved_upcall_mask = (_s_)->saved_upcall_mask; \
    (_d_)->__DECL_REG_LO16(flags) = (_s_)->__DECL_REG_LO16(flags); \
    (_d_)->__DECL_REG_LO16(sp) = (_s_)->__DECL_REG_LO16(sp); \
    (_d_)->ss = (_s_)->ss; \
    (_d_)->es = (_s_)->es; \
    (_d_)->ds = (_s_)->ds; \
    (_d_)->fs = (_s_)->fs; \
    (_d_)->gs = (_s_)->gs; \
} while (0)

registers are copied, but additional fields (error_code, entry_vector,
saved_upcall_mask).
Maybe you could try to copy them again.
Also, just to be on the safe side, you could memset the structure
where "Backing memory is pre-zeroed" comments are.

Frediano



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.