Re: early crash while loading dom0 kernel between git:19730dbb3f and git:414dde38b0

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 16/02/2025 11:09 pm, Greg A. Woods wrote:
> I had been testing 4.20-rc (at git:19730dbb3f) relatively successfully
> on a older Dell PE-2950 server.
>
> Today I tried upgrading to git:414dde38b0 and I encountered the
> following highly repeatable crash on first boot.  (note the git commit
> shown in the log is from my(robohack) local NetBSD patches on GitHub,
> none of which are in the Xen kernel itself -- just tools)
>
> Note the dom0 kernel is NetBSD (-current as of about a year ago).
>
> I'm not on the xen-devel list, so please email me directly (also see
> alternate addresses in my signature below).  Note I can't even send
> from my normal email because inumbo.net <http://inumbo.net/> don't understand 
> the rules of
> the DNS and have botched the MX records for xenproject.org 
> <http://xenproject.org/>.
>
>
> The offending address ([<ffff82d040201015>] R _stextentry+0x15/0x165)
> seems to be found here (according to "objdump -S xen-syms"):
>
> ffff82d040201000 <restore_all_guest>:
>
>         .section .text.entry, "ax", @progbits
>
> /* %rbx: struct vcpu, interrupts disabled */
> FUNC_LOCAL(restore_all_guest)
>         ASSERT_INTERRUPTS_DISABLED
> ffff82d040201000:       9c                      pushfq
> ffff82d040201001:       f6 44 24 01 02          testb  $0x2,0x1(%rsp)
> ffff82d040201006:       74 02                   je     ffff82d04020100a 
> <restore_all_guest+
> 0xa>
> ffff82d040201008:       0f 0b                   ud2
> ffff82d04020100a:       48 83 c4 08             add    $0x8,%rsp
>
>         /* Stash guest SPEC_CTRL value while we can read struct vcpu. */
>         mov VCPU_arch_msrs(%rbx), %rdx
> ffff82d04020100e:       48 8b 93 68 0d 00 00    mov    0xd68(%rbx),%rdx
>         mov VCPUMSR_spec_ctrl_raw(%rdx), %r15d
> ffff82d040201015:       44 8b 3a                mov    (%rdx),%r15d
>
>
> If I'm not mistaken this is from xen/arch/x86/x86_64/entry.S.  I don't
> see any recent changes there though, so I'm not sure where to go from
> here.  Did something deeper in struct vcpu change?

We see things like this very occasionally when doing `git pull &&
make`.  There's something that seems to not regenerate asm-offsets
properly on an incremental build, but I've never been able to reproduce
enough to debug.

There are definitely changes in struct domain in the range you give. 
There's not obviously changes to struct vcpu.

Nevertheless, can you try a clean build to start with, and see if that
changes things?

> Start @ 0x200000 [1=0x619000-0x6190ec]...
> Xen 4.20-rc
> (XEN) [000000341c1f78e9] Xen version 4.20-rc (woods@.local) (gcc (nb1 
> 20200907) 9.3.0) debug=y Sun Feb 16 13:33:02 PST 2025
> (XEN) [000000341e299905] Latest ChangeSet: Wed Jan 29 13:29:16 2025 -0800 
> git:72eea1d3cb-dirty
> (XEN) [000000341fba9b9d] build-id: 0e6a2491c4ad94bdf27ff33fcefc31b5a8b61784
> (XEN) [0000003420fc47e1] CPU Vendor: Intel, Family 6 (0x6), Model 23 (0x17), 
> Stepping 6 (raw 00010676)
> (XEN) [0000003422aea44d] BSP microcode revision: 0x0000060f
> (XEN) [0000003423ad77fc] Bootloader: NetBSD/x86 BIOS Boot, Revision 5.11 (Sun 
> Dec  8 23:54:34 UTC 2024) (from NetBSD 9.99.81)
> (XEN) [0000003425c00815] Command line: loglvl=all bootscrub=false 
> dom0=pv,verbose=1 dom0_mem=20G console=com1,vga console_timestamps=datems 
> dom0_max_vcpus=8 dom0_vcpus_pin=true guest_loglvl=all pv-l1tf=off,domu=off 
> cpuid=rdrand vpmu=on,ipc spec-ctrl=no

A couple of unrelated notes.  Boolean options can be written without
suffixes, so `dom0=pv,verbose` suffices, as does `dom0_vcpus_pin`.

You really want to use dom0_max_vcpus=1-8 instead of just plain 8.

Also, spec-ctrl=no also disables pv-l1tf (both parts) so you don't need
to specify that separately.

~Andrew