[Xen-devel] xen-hypervisor-4.3-amd64 4.3.0-3+b1 -> Intel M/B, I/O virt. enabled, start vm -> Kernel panic

[Jo Mills <jo@xxxxxxxxxxxxxxxx>]

Hi,

I started with a thread on xen-users and was asked by Ian Campbell if I 
would take more logs and raise the issue with xen-devel.  Please see 
thread with subject "Create domU with pciback fails, then my system 
re-boots! xen-hypervisor-4.3-amd64" on xen-users for more background 
if you want it.  Below I have to tried to list the steps I have taken 
to reproduce the problem.


[1] Machine "green"

        Intel S32000SHV motherboard
        8 GB RAM
        GenuineIntel Intel(R) Core(TM)2 Quad CPU
        BIOS version S3200X38.86B.00.00.52

    (Machine blue is the same hardware - I use them as a pair of DRBD 
     connected hosts).

        
[2] xen and linux version on green:

        dom0 is Debian Jessie:

    Linux version 3.13-1-amd64 (debian-kernel@xxxxxxxxxxxxxxxx) \
        (gcc version 4.8.2 (Debian 4.8.2-16) ) \
            #1 SMP Debian 3.13.10-1 (2014-04-15)

    xen-hypervisor-4.3-amd64            4.3.0-3+b1
    xen-system-amd64                    4.3.0-3+b1
    xen-tools                           4.4-1     
    xen-utils-4.3                       4.3.0-3+b1
    xen-utils-common                    4.3.0-3   
    xenstore-utils                      4.3.0-3+b1


    system is up to date from Jessie as of Sat 17 May 16:58:01 BST 2014
    

[3] grub2 configured as:

    GRUB_CMDLINE_XEN="dom0_mem=2G,max:2G dom0_max_vcpus=1 \
        dom0_vcpus_pin vga=gfx-1280x1024x16 noreboot loglvl=all \
            guest_loglvl=all com1=115200,8n1,0x3f8,4 console=com1,vga"
    GRUB_CMDLINE_LINUX="console=hvc0 earlyprintk=xen"

    null modem cable connected to screen session on my desktop PC.
    

[4] There are four ethernet devices fitted, and after the various udev
    renaming of Ethernet interfaces these come out as being:

    eth0 via-rhine 0000:04:00.0 assigned for zone LOC xenbr0

    eth1 via-rhine 0000:04:01.0 assigned for zone DMZ (pci-passthrough)

    eth2 e1000 0000:04:02.0 used for DBRB

    eth3 e1000e 0000:01:00.0 planned for Windows client domU


[5] I have patched /etc/xen/scripts/block-drbd as per 
    http://lists.xen.org/archives/html/xen-devel/2014-02/msg01190.html 
    so there are two case statements modifed to be:
    
            case $t in 
                drbd|phy)
                drbd_resource=$p
                drbd_role="$(/sbin/drbdadm role $drbd_resource)"
                    .
                    .
                    .
                    
    and
    
        remove)
            case $t in 
                drbd|phy)
                p=$(xenstore_read "$XENBUS_PATH/params")
                drbd_resource=$p
                    .
                    .
                    .



[6] Start test with the following BIOS options all disbaled:

        Intel(R) Virtualization Technology
        Intel(R) VT for Directed I/O
        Multi-Thread Support In MPS table
        Execute Disable Bit


    Re-boot green from console, login via SSH from desktop machine.
    
    ~#  xl pci-assignable-list
    0000:01:00.0
    0000:04:01.0

    so these two cards are available for pci passthrough.  DRBD is 
    working fine with machine "blue" (which is running wheezy and uses 
    xm). 
    
    On blue, shutdown vm-server-21 and when it has gone, start it on 
    machine green, this vm uses device 04:01.0.
    
    The vm starts as expected and DRBD is happy.
    
    ~# xl list          
    Name                ID   Mem VCPUs      State   Time(s)
    Domain-0             0  2046     1     r-----      18.4
    vm-server-21         1  1020     1     -b----      16.9

    Shutdown this vm on green and re-start it on blue.  Logout of SSH 
    session and re-start green from console.  Go into BIOS and enable:
    
        Intel(R) Virtualization Technology
        Intel(R) VT for Directed I/O
    
    login via SSH from desktop machine.
    
    ~# xl dmesg 
    
        contains    
        
    (XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
    (XEN) traps.c:3061: GPF (0000): ffff82c4c02772f8 -> ffff82c4c0218927
    (XEN) Intel VT-d Snoop Control not enabled.
    (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
    (XEN) Intel VT-d Queued Invalidation not enabled.
    (XEN) Intel VT-d Interrupt Remapping not enabled.
    (XEN) Intel VT-d Shared EPT tables not enabled.
    (XEN) I/O virtualisation enabled

        so I am hopeful that at some time I might get a windows domU 
        to run (yet "grep vmx /proc/cpuinfo" returns nothing, so maybe 
        it won't work).  As before, the same two ethernet cards are 
        available via passthrough.
        
    ~#  xl pci-assignable-list
    0000:01:00.0
    0000:04:01.0
    
    
    If I now shutdown vm-server-21 on node blue, and re-start it green 

~# xl -vvv create -c /etc/xen/vm-server-21.cfg
Parsing config from /etc/xen/vm-server-21.cfg
libxl: debug: libxl_create.c:1230:do_domain_create: ao 0x19671f0: create: 
how=(nil) callback=(nil) poller=0x1966ba0
libxl: verbose: libxl_create.c:130:libxl__domain_build_info_setdefault: 
qemu-xen is unavailable, use qemu-xen-traditional instead: No such file or 
directory
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvdb 
spec.backend=unknown
libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=xvdb, uses 
script=... assuming phy backend
libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk 
vdev=xvdb, using backend phy
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda 
spec.backend=unknown
libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk 
vdev=xvda, using backend phy
libxl: debug: libxl_create.c:675:initiate_domain_create: running bootloader
libxl: debug: libxl_bootloader.c:327:libxl__bootloader_run: no bootloader 
configured, using user supplied kernel
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch 
w=0x1967608: deregister unregistered
libxl: debug: libxl_x86.c:82:e820_sanitize: Memory: 1048576kB End of RAM: 
0x40000 (PFN) Delta: 0kB, PCI start: 3665832kB (0xdfbea PFN), Balloon 0kB

libxl: debug: libxl_x86.c:201:e820_sanitize: :  [0 -> 40000] RAM
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [40000 -> dfbea] Unusable
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [dfbea -> dfc96] ACPI NVS
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [dfc96 -> dfcfa] Unusable
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [dfcfa -> dfd5f] Reserved
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [dfd5f -> dfd69] Unusable
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [dfd69 -> dfddf] ACPI NVS
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [dfddf -> dfde5] Unusable
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [dfde5 -> dfdff] ACPI
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [dfdff -> dfe00] Unusable
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [dfe00 -> dff00] Reserved
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [f0000 -> f4000] Reserved
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [fee00 -> fee01] Reserved
libxl: debug: libxl_x86.c:201:e820_sanitize: :  [fff80 -> fff8c] Reserved
domainbuilder: detail: xc_dom_allocate: cmdline="root=/dev/xvdb ro xencons=tty 
swiotlb=force", features="(null)"
libxl: debug: libxl_dom.c:341:libxl__build_pv: pv kernel mapped 0 path 
/boot/vmlinuz-2.6.26-2-xen-amd64

domainbuilder: detail: xc_dom_kernel_file: 
filename="/boot/vmlinuz-2.6.26-2-xen-amd64"
domainbuilder: detail: xc_dom_malloc_filemap    : 1666 kB
domainbuilder: detail: xc_dom_malloc            : 7801 kB
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x1a0b72 -> 0x79e530
domainbuilder: detail: xc_dom_ramdisk_file: 
filename="/boot/initrd.img-2.6.26-2-xen-amd64"
domainbuilder: detail: xc_dom_malloc_filemap    : 7926 kB
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.3, caps xen-3.0-x86_64 
xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ... 
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ... 
domainbuilder: detail: xc_dom_probe_bzimage_kernel: kernel is not a bzImage
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying ELF-generic loader ... 
domainbuilder: detail: loader probe OK
xc: detail: elf_parse_binary: phdr: paddr=0x200000 memsz=0x2fe000
xc: detail: elf_parse_binary: phdr: paddr=0x4fe000 memsz=0x529a8
xc: detail: elf_parse_binary: phdr: paddr=0x551000 memsz=0x888
xc: detail: elf_parse_binary: phdr: paddr=0x552000 memsz=0xdf918
xc: detail: elf_parse_binary: memory: 0x200000 -> 0x631918
xc: detail: elf_xen_parse_note: GUEST_OS = "linux"
xc: detail: elf_xen_parse_note: GUEST_VERSION = "2.6"
xc: detail: elf_xen_parse_note: XEN_VERSION = "xen-3.0"
xc: detail: elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
xc: detail: elf_xen_parse_note: PADDR_OFFSET = 0x0
xc: detail: elf_xen_parse_note: ENTRY = 0xffffffff80200000
xc: detail: elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff80208000
xc: detail: elf_xen_parse_note: unknown xen elf note (0xd)
xc: detail: elf_xen_parse_note: FEATURES = 
"writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel"
xc: detail: elf_xen_parse_note: LOADER = "generic"
xc: detail: elf_xen_parse_note: SUSPEND_CANCEL = 0x1
xc: detail: elf_xen_addr_calc_check: addresses:
xc: detail:     virt_base        = 0xffffffff80000000
xc: detail:     elf_paddr_offset = 0x0
xc: detail:     virt_offset      = 0xffffffff80000000
xc: detail:     virt_kstart      = 0xffffffff80200000
xc: detail:     virt_kend        = 0xffffffff80631918
xc: detail:     virt_entry       = 0xffffffff80200000
xc: detail:     p2m_base         = 0xffffffffffffffff
domainbuilder: detail: xc_dom_parse_elf_kernel: xen-3.0-x86_64: 
0xffffffff80200000 -> 0xffffffff80631918
domainbuilder: detail: xc_dom_mem_init: mem 1024 MB, pages 0x40000 pages, 4k 
each
domainbuilder: detail: xc_dom_mem_init: 0x40000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: x86_compat: guest xen-3.0-x86_64, address size 64
domainbuilder: detail: xc_dom_malloc            : 2048 kB
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_alloc_segment:   kernel       : 
0xffffffff80200000 -> 0xffffffff80632000  (pfn 0x200 + 0x432 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 
0x200+0x432 at 0x7f6b8b0cc000
xc: detail: elf_load_binary: phdr 0 at 0x7f6b8b0cc000 -> 0x7f6b8b3ca000
xc: detail: elf_load_binary: phdr 1 at 0x7f6b8b3ca000 -> 0x7f6b8b41c9a8
xc: detail: elf_load_binary: phdr 2 at 0x7f6b8b41d000 -> 0x7f6b8b41d888
xc: detail: elf_load_binary: phdr 3 at 0x7f6b8b41e000 -> 0x7f6b8b45c6b0
domainbuilder: detail: xc_dom_alloc_segment:   ramdisk      : 
0xffffffff80632000 -> 0xffffffff81d14000  (pfn 0x632 + 0x16e2 pages)
domainbuilder: detail: xc_dom_malloc            : 137 kB
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 
0x632+0x16e2 at 0x7f6b899ea000
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x7bd8b0 -> 0x16e1610
domainbuilder: detail: xc_dom_alloc_segment:   phys2mach    : 
0xffffffff81d14000 -> 0xffffffff81f14000  (pfn 0x1d14 + 0x200 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 
0x1d14+0x200 at 0x7f6b897ea000
domainbuilder: detail: xc_dom_alloc_page   :   start info   : 
0xffffffff81f14000 (pfn 0x1f14)
domainbuilder: detail: xc_dom_alloc_page   :   xenstore     : 
0xffffffff81f15000 (pfn 0x1f15)
domainbuilder: detail: xc_dom_alloc_page   :   console      : 
0xffffffff81f16000 (pfn 0x1f16)
domainbuilder: detail: nr_page_tables: 0x0000ffffffffffff/48: 
0xffff000000000000 -> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x0000007fffffffff/39: 
0xffffff8000000000 -> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x000000003fffffff/30: 
0xffffffff80000000 -> 0xffffffffbfffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x00000000001fffff/21: 
0xffffffff80000000 -> 0xffffffff81ffffff, 16 table(s)
domainbuilder: detail: xc_dom_alloc_segment:   page tables  : 
0xffffffff81f17000 -> 0xffffffff81f2a000  (pfn 0x1f17 + 0x13 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 
0x1f17+0x13 at 0x7f6b8e36f000
domainbuilder: detail: xc_dom_alloc_page   :   boot stack   : 
0xffffffff81f2a000 (pfn 0x1f2a)
domainbuilder: detail: xc_dom_build_image  : virt_alloc_end : 0xffffffff81f2b000
domainbuilder: detail: xc_dom_build_image  : virt_pgtab_end : 0xffffffff82000000
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: arch_setup_bootearly: doing nothing
domainbuilder: detail: xc_dom_compat_check: supported guest type: 
xen-3.0-x86_64 <= matches
domainbuilder: detail: xc_dom_compat_check: supported guest type: 
xen-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32
domainbuilder: detail: xc_dom_compat_check: supported guest type: 
hvm-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64
domainbuilder: detail: xc_dom_update_guest_p2m: dst 64bit, pages 0x40000
domainbuilder: detail: clear_page: pfn 0x1f16, mfn 0x198307
domainbuilder: detail: clear_page: pfn 0x1f15, mfn 0x198308
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x1f14+0x1 
at 0x7f6b8e550000
domainbuilder: detail: start_info_x86_64: called
domainbuilder: detail: setup_hypercall_page: vaddr=0xffffffff80208000 pfn=0x208
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail:    allocated
domainbuilder: detail:       malloc             : 10027 kB
domainbuilder: detail:       anon mmap          : 0 bytes
domainbuilder: detail:    mapped
domainbuilder: detail:       file mmap          : 9593 kB
domainbuilder: detail:       domU mmap          : 29856 kB
domainbuilder: detail: arch_setup_bootlate: shared_info: pfn 0x0, mfn 0xdfc99
domainbuilder: detail: shared_info_x86_64: called
domainbuilder: detail: vcpu_x86_64: called
domainbuilder: detail: vcpu_x86_64: cr3: pfn 0x1f17 mfn 0x198306
domainbuilder: detail: launch_vm: called, ctxt=0x7fff4af4aee0
domainbuilder: detail: xc_dom_release: called
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvdb 
spec.backend=phy
libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=xvdb, uses 
script=... assuming phy backend
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x1968528 
wpath=/local/domain/0/backend/vbd/1/51728/state token=3/0: register slotnum=3
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda 
spec.backend=phy
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x196fb78 
wpath=/local/domain/0/backend/vbd/1/51712/state token=2/1: register slotnum=2
libxl: debug: libxl_create.c:1243:do_domain_create: ao 0x19671f0: inprogress: 
poller=0x1966ba0, flags=i
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1968528 
wpath=/local/domain/0/backend/vbd/1/51728/state token=3/0: event 
epath=/local/domain/0/backend/vbd/1/51728/state
libxl: debug: libxl_event.c:647:devstate_watch_callback: backend 
/local/domain/0/backend/vbd/1/51728/state wanted state 2 still waiting state 1
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x196fb78 
wpath=/local/domain/0/backend/vbd/1/51712/state token=2/1: event 
epath=/local/domain/0/backend/vbd/1/51712/state
libxl: debug: libxl_event.c:647:devstate_watch_callback: backend 
/local/domain/0/backend/vbd/1/51712/state wanted state 2 still waiting state 1
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1968528 
wpath=/local/domain/0/backend/vbd/1/51728/state token=3/0: event 
epath=/local/domain/0/backend/vbd/1/51728/state
libxl: debug: libxl_event.c:643:devstate_watch_callback: backend 
/local/domain/0/backend/vbd/1/51728/state wanted state 2 ok
libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x1968528 
wpath=/local/domain/0/backend/vbd/1/51728/state token=3/0: deregister slotnum=3
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch 
w=0x1968528: deregister unregistered
libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: 
/etc/xen/scripts/block-drbd add
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x196fb78 
wpath=/local/domain/0/backend/vbd/1/51712/state token=2/1: event 
epath=/local/domain/0/backend/vbd/1/51712/state
libxl: debug: libxl_event.c:643:devstate_watch_callback: backend 
/local/domain/0/backend/vbd/1/51712/state wanted state 2 ok
libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x196fb78 
wpath=/local/domain/0/backend/vbd/1/51712/state token=2/1: deregister slotnum=2
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch 
w=0x196fb78: deregister unregistered
libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: 
/etc/xen/scripts/block add

    and it just sits here.


    I get the following at the end of the serial console log:
    
    [  498.336430] ---[ end trace b9630577ecf84cd8 ]---
    [  498.340053] Kernel panic - not syncing: Fatal exception in interrupt
    (XEN) Domain 0 crashed: 'noreboot' set - not rebooting.

    and that's "green" now stuck.  It also "did something horrible" to 
    node blue as there all my vm's seem to have vanished - but that's 
    a different problem.  Re-boot both green and blue, re-start vm's 
    on node blue which seems OK now, DRBD is also happy.
    
    Serial console log for node green attached as 
    screenlog.0.serial_console_log.tgz.
    
    If there is anything more I can add, please let me know and I'll 
    try my best to get more information.

Best regards,

Jo.

Attachment: screenlog.0.serial_console_log.tgz
Description: application/gtar-compressed

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel