|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: domU crash when resuming from suspend - Xen 4.22
On Sun, Jul 19, 2026 at 10:04:42PM +0100, Frediano Ziglio wrote: > On Sun, 19 Jul 2026 at 20:40, Marek Marczykowski-Górecki > <marmarek@xxxxxxxxxxxxxxxxxxxxxx> wrote: > > > > On Sun, Jul 19, 2026 at 08:31:28AM +0100, Frediano Ziglio wrote: > > > On Sun, 19 Jul 2026 at 03:00, Marek Marczykowski-Górecki > > > <marmarek@xxxxxxxxxxxxxxxxxxxxxx> wrote: > > > > > > > > On Sat, Jul 18, 2026 at 04:03:29PM +0200, Marek Marczykowski-Górecki > > > > wrote: > > > > > On Fri, Jul 17, 2026 at 09:22:47PM +0100, Frediano Ziglio wrote: > > > > > > On Tue, 14 Jul 2026 at 18:12, Marek Marczykowski-Górecki > > > > > > <marmarek@xxxxxxxxxxxxxxxxxxxxxx> wrote: > > > > > > > > > > > > > > On Fri, Jun 19, 2026 at 01:24:09PM +0200, Marek > > > > > > > Marczykowski-Górecki wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > I'm continuing testing 4.22 and found this: > > > > > > > > > > > > > > > > When a domU is suspended (even without host suspend) it crashes > > > > > > > > on > > > > > > > > resume with: > > > > > > > > > > > > > > > > [ 24.374629] BUG: kernel NULL pointer dereference, > > > > > > > > address: 0000000000000020 > > > > > > > > [ 24.374647] #PF: supervisor read access in kernel mode > > > > > > > > [ 24.374657] #PF: error_code(0x0000) - not-present page > > > > > > > > [ 24.374668] PGD 0 P4D 0 > > > > > > > > [ 24.374675] Oops: Oops: 0000 [#1] SMP NOPTI > > > > > > > > [ 24.374685] CPU: 0 UID: 0 PID: 19 Comm: migration/0 Not > > > > > > > > tainted 6.18.31-1.qubes.13.fc41.x86_64 #1 PREEMPT(full) > > > > > > > > [ 24.374705] Stopper: multi_cpu_stop+0x0/0x140 <- > > > > > > > > __stop_cpus.constprop.0+0x6f/0xb0 > > > > > > > > [ 24.374723] RIP: 0010:multi_cpu_stop+0x63/0x140 > > > > > > > > [ 24.374734] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 > > > > > > > > 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd > > > > > > > > 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 > > > > > > > > 89 77 24 41 89 47 20 41 83 fd > > > > > > > > [ 24.374764] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046 > > > > > > > > [ 24.374775] RAX: 0000000000000000 RBX: ffffc90000a13de4 > > > > > > > > RCX: 0000000000000404 > > > > > > > > [ 24.374790] RDX: 0000000000000040 RSI: ffffffffffffffff > > > > > > > > RDI: 0000000000770ef0 > > > > > > > > [ 24.374801] RBP: 0000000000000002 R08: ffffc900000abc80 > > > > > > > > R09: ffffc900000abc8c > > > > > > > > [ 24.374813] R10: 0000000000000000 R11: ffff88801e81cdc0 > > > > > > > > R12: ffffffff81835ac0 > > > > > > > > [ 24.374825] R13: 0000000000000000 R14: 0000000000000000 > > > > > > > > R15: 0000000000000000 > > > > > > > > [ 24.374837] FS: 0000000000000000(0000) > > > > > > > > GS:ffff88809b369000(0000) knlGS:0000000000000000 > > > > > > > > [ 24.374850] CS: 0010 DS: 0000 ES: 0000 CR0: > > > > > > > > 0000000080050033 > > > > > > > > [ 24.374860] CR2: 0000000000000020 CR3: 000000000d6a2002 > > > > > > > > CR4: 0000000000770ef0 > > > > > > > > [ 24.374873] PKRU: 00000000 > > > > > > > > [ 24.374877] Call Trace: > > > > > > > > [ 24.374884] <TASK> > > > > > > > > [ 24.374891] ? __pfx_multi_cpu_stop+0x10/0x10 > > > > > > > > [ 24.374900] cpu_stopper_thread+0xa3/0x170 > > > > > > > > [ 24.374908] ? __pfx_smpboot_thread_fn+0x10/0x10 > > > > > > > > [ 24.374917] smpboot_thread_fn+0xf3/0x220 > > > > > > > > [ 24.374925] kthread+0xfc/0x240 > > > > > > > > [ 24.374933] ? __pfx_kthread+0x10/0x10 > > > > > > > > [ 24.374940] ? __pfx_kthread+0x10/0x10 > > > > > > > > [ 24.374949] ret_from_fork+0x158/0x170 > > > > > > > > [ 24.374957] ? __pfx_kthread+0x10/0x10 > > > > > > > > [ 24.374965] ret_from_fork_asm+0x1a/0x30 > > > > > > > > [ 24.374973] </TASK> > > > > > > > > [ 24.374978] Modules linked in: snd_seq_dummy snd_hrtimer > > > > > > > > snd_seq snd_seq_device snd_timer snd soundcore cfg80211 rfkill > > > > > > > > vfat fat xenfs binfmt_misc nft_reject_inet nf_reject_ipv4 > > > > > > > > nf_reject_ipv6 nft_reject nft_ct nf_conntrack nf_defrag_ipv6 > > > > > > > > nf_defrag_ipv4 nf_tables intel_rapl_msr intel_rapl_common > > > > > > > > intel_uncore_frequency_common intel_pmc_ssram_telemetry > > > > > > > > intel_vsec polyval_clmulni ghash_clmulni_intel xen_netfront > > > > > > > > xen_privcmd xen_gntdev xen_gntalloc xen_blkback xen_evtchn fuse > > > > > > > > loop nfnetlink ip_tables overlay xen_blkfront > > > > > > > > [ 24.375064] CR2: 0000000000000020 > > > > > > > > [ 24.375071] ---[ end trace 0000000000000000 ]--- > > > > > > > > [ 24.375080] RIP: 0010:multi_cpu_stop+0x63/0x140 > > > > > > > > [ 24.375089] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 > > > > > > > > 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd > > > > > > > > 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 > > > > > > > > 89 77 24 41 89 47 20 41 83 fd > > > > > > > > [ 24.375115] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046 > > > > > > > > [ 24.375124] RAX: 0000000000000000 RBX: ffffc90000a13de4 > > > > > > > > RCX: 0000000000000404 > > > > > > > > [ 24.375142] RDX: 0000000000000040 RSI: ffffffffffffffff > > > > > > > > RDI: 0000000000770ef0 > > > > > > > > [ 24.375155] RBP: 0000000000000002 R08: ffffc900000abc80 > > > > > > > > R09: ffffc900000abc8c > > > > > > > > [ 24.375170] R10: 0000000000000000 R11: ffff88801e81cdc0 > > > > > > > > R12: ffffffff81835ac0 > > > > > > > > [ 24.375184] R13: 0000000000000000 R14: 0000000000000000 > > > > > > > > R15: 0000000000000000 > > > > > > > > [ 24.375200] FS: 0000000000000000(0000) > > > > > > > > GS:ffff88809b369000(0000) knlGS:0000000000000000 > > > > > > > > [ 24.375214] CS: 0010 DS: 0000 ES: 0000 CR0: > > > > > > > > 0000000080050033 > > > > > > > > [ 24.375226] CR2: 0000000000000020 CR3: 000000000d6a2002 > > > > > > > > CR4: 0000000000770ef0 > > > > > > > > [ 24.375241] PKRU: 00000000 > > > > > > > > [ 24.375247] Kernel panic - not syncing: Fatal exception > > > > > > > > [ 25.417403] Shutting down cpus with NMI > > > > > > > > [ 25.417447] Kernel Offset: disabled > > > > > > > > > > > > > > > > There is no change on the Linux kernel side (neither dom0 nor > > > > > > > > domU), the > > > > > > > > only change is updating Xen 4.19.4 -> 4.22-rc2 (seems to be > > > > > > > > already > > > > > > > > broken on -rc1 too). This happens for both HVM and PVH domains. > > > > > > > > Haven't > > > > > > > > tried PV domU (but dom0 survives host suspend). > > > > > > > > > > > > > > Any ideas? > > > > > > > > > > > > > > > > > > > Hi, > > > > > > not many ideas but while testing my libxenguest changes I found > > > > > > similar failures using xl create/save/restore. But I was testing > > > > > > more > > > > > > PV guests. > > > > > > Does 4.21 work ? > > > > > > > > > > It seems 4.21 also has this issue: > > > > > https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/15414211523 > > > > > But 4.20 is okay: > > > > > https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/15414227095 > > > > > > > > > > I guess I can bisect it further. > > > > > > > > So, bisect says the first bad commit is: > > > > > > > > commit 9f892f84c27970840d255aef7e2851b762967de7 > > > > Author: Andrew Cooper <andrew.cooper3@xxxxxxxxxx> > > > > Date: Mon Dec 30 11:49:14 2024 +0000 > > > > > > > > x86/domctl: Stop using XLAT_cpu_user_regs() > > > > > > > > My test isn't 100% reliable, but I retested parent of this commit a > > > > couple of times and it was good every time. > > > > > > > > I wanted to test reverting it on top of 4.21, but there is a lot of > > > > FRED-related work on top that makes such revert quite hard, to say the > > > > least... > > > > > > > > > > The generated XLAT_cpu_user_regs was > > > > > > #define XLAT_cpu_user_regs(_d_, _s_) do { \ > > > (_d_)->__DECL_REG_LO8(b) = (_s_)->__DECL_REG_LO8(b); \ > > > (_d_)->__DECL_REG_LO8(c) = (_s_)->__DECL_REG_LO8(c); \ > > > (_d_)->__DECL_REG_LO8(d) = (_s_)->__DECL_REG_LO8(d); \ > > > (_d_)->__DECL_REG_LO16(si) = (_s_)->__DECL_REG_LO16(si); \ > > > (_d_)->__DECL_REG_LO16(di) = (_s_)->__DECL_REG_LO16(di); \ > > > (_d_)->__DECL_REG_LO16(bp) = (_s_)->__DECL_REG_LO16(bp); \ > > > (_d_)->__DECL_REG_LO8(a) = (_s_)->__DECL_REG_LO8(a); \ > > > (_d_)->error_code = (_s_)->error_code; \ > > > (_d_)->entry_vector = (_s_)->entry_vector; \ > > > (_d_)->__DECL_REG_LO16(ip) = (_s_)->__DECL_REG_LO16(ip); \ > > > (_d_)->cs = (_s_)->cs; \ > > > (_d_)->saved_upcall_mask = (_s_)->saved_upcall_mask; \ > > > (_d_)->__DECL_REG_LO16(flags) = (_s_)->__DECL_REG_LO16(flags); \ > > > (_d_)->__DECL_REG_LO16(sp) = (_s_)->__DECL_REG_LO16(sp); \ > > > (_d_)->ss = (_s_)->ss; \ > > > (_d_)->es = (_s_)->es; \ > > > (_d_)->ds = (_s_)->ds; \ > > > (_d_)->fs = (_s_)->fs; \ > > > (_d_)->gs = (_s_)->gs; \ > > > } while (0) > > > > > > registers are copied, but additional fields (error_code, entry_vector, > > > saved_upcall_mask). > > > Maybe you could try to copy them again. > > > Also, just to be on the safe side, you could memset the structure > > > where "Backing memory is pre-zeroed" comments are. > > > > This does not help :( > > > > Specifically: > > 1. Applied on top of staging-4.21: > > - build: > > https://gitlab.com/xen-project/people/marmarek/xen/-/commits/188ca612c51a57ef54b917c833eaeb59666994c9 > > - test (fails): > > https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/2688813821 > > 2. Applied directly on top of 9f892f84c27970840d255aef7e2851b762967de7: > > - build: > > https://gitlab.com/xen-project/people/marmarek/xen/-/commits/dd3dee35fc4fcd6ba284f3f5fa78b333c472888e > > - test (fails): > > https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/2688832398 > > > > Yet, parent of 9f892f84c27970840d255aef7e2851b762967de7 still passes: > > - build: > > https://gitlab.com/xen-project/people/marmarek/xen/-/commits/dbcbbed4e9dc25faa211d359c2f04a9c70f087c9 > > - test (pass): > > https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/2688835512 > > > > Hint about reading the above: I use two branches, one for building > > (branch "test") and one for running the test (branch > > "automation-uki-grub-only"), > > which takes build artifacts from the build branch. I do that to not > > rebase test scripts. It's a bit opaque, but the console log of the test > > contains "Latest ChangeSet" fields telling what actually was tested. > > > > If it does not take long to test you could revert hunk by hunk > starting from xen/include/xlat.lst. Ok, I found what is wrong. It isn't about XLAT branch, but the native one, x86_64 has a couple more registers... This fixes it: diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c index 9af833b6da59f66d58eac3369c30a89919d0d805..893ebaa13a6f934e1af026f9eb13b1d18f746214 100644 --- a/xen/arch/x86/domain.c +++ b/xen/arch/x86/domain.c @@ -1280,6 +1280,14 @@ int arch_set_info_guest( if ( !compat ) { + v->arch.user_regs.r15 = c.nat->user_regs.r15; + v->arch.user_regs.r14 = c.nat->user_regs.r14; + v->arch.user_regs.r13 = c.nat->user_regs.r13; + v->arch.user_regs.r12 = c.nat->user_regs.r12; + v->arch.user_regs.r11 = c.nat->user_regs.r11; + v->arch.user_regs.r10 = c.nat->user_regs.r10; + v->arch.user_regs.r9 = c.nat->user_regs.r9; + v->arch.user_regs.r8 = c.nat->user_regs.r8; v->arch.user_regs.rbx = c.nat->user_regs.rbx; v->arch.user_regs.rcx = c.nat->user_regs.rcx; v->arch.user_regs.rdx = c.nat->user_regs.rdx; diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c index b8bc0caaf205828f9254e03584709482f3ba38a6..40f4cb049ce3f84ef38a604254392548a41342de 100644 --- a/xen/arch/x86/domctl.c +++ b/xen/arch/x86/domctl.c @@ -1440,6 +1440,14 @@ void arch_get_info_guest(struct vcpu *v, vcpu_guest_context_u c) if ( !compat ) { /* Backing memory is pre-zeroed. */ + c.nat->user_regs.r15 = v->arch.user_regs.r15; + c.nat->user_regs.r14 = v->arch.user_regs.r14; + c.nat->user_regs.r13 = v->arch.user_regs.r13; + c.nat->user_regs.r12 = v->arch.user_regs.r12; + c.nat->user_regs.r11 = v->arch.user_regs.r11; + c.nat->user_regs.r10 = v->arch.user_regs.r10; + c.nat->user_regs.r9 = v->arch.user_regs.r9; + c.nat->user_regs.r8 = v->arch.user_regs.r8; c.nat->user_regs.rbx = v->arch.user_regs.rbx; c.nat->user_regs.rcx = v->arch.user_regs.rcx; c.nat->user_regs.rdx = v->arch.user_regs.rdx; -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab Attachment:
signature.asc
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |