Re: [Xen-users] Problem with VT-d on Mainboard Intel DQ45CB

["Andrew Lyon" <andrew.lyon@xxxxxxxxx>]

On Mon, Oct 27, 2008 at 11:04 AM, Franz Regensburger
<Franz.Regensburger@xxxxxxxx> wrote:
> Hello Todd,
>
> thank you for your comments.
> I did not have time yet to follow your suggestions (I certainly will).
>> To get to the point where you can be helpful, you will need to get the
>> latest xen-unstable.
>>
>> hg clone http://xenbits.xensource.com/xen-unstable.hg
>> hg clone http://xenbits.xensource.com/linux-2.6.18-xen.hg
>> cd xen-unstable
>> make world
>
> However, yesterday I was able to make some progress with respect to the Intel 
> board DQ45CB that
> solves most of my problems.
>
> It seems that most problems were caused by the board and not Xen!
> I think it is important to share my new experience as early as possible in 
> order to prevent others for
> suffering in vain or doing unneccessary error analysis.
>
> Some days ago Intel put a new firmware for the board onto their WEB-site
>        CBQ4510H.86A.0061.BI.ZIP
>
> I upgraded my board from CBQ4510H.86A.0059 with that new BIOS and the 
> following problems seem to have vanished:
>
> Summary:
> ========
> Update to BIOS CBQ4510H.86A.0061.BI.ZIP solved the following issues:
>
> - Whenever I boot my Xen 2.6.18.8 (Xen 3.3) with kernel parameter iommu=1
> the boot process freezes.
> SOLVED (details below)
>
> and related to that
>
> - VT-d broken. No PCI backend hiding with HVM guest possible
> SOLVED (details below)
>
> - When I insert a PCIe x16 Graphics card (ATI Radeon HD 2400 Pro) the BIOS 
> menu display
> somehow becomes slow. The experience is like if you use a 300 Baut serial 
> line connection.
> SOLVED (details below)
>
> - PCIe x16 graphics with Xen kernel broken
> SOLVED (details below)
>
>
> I will also post the above short note on xen-devel
>
> Regards,
>
> Franz
>
> ##############################
>
> Details:
>
> My current setup:
> Xen Source: http://bits.xensource.com/oss-xen/release/3.3.0/xen-3.3.0.tar.gz
> Xen Kernel: hg clone http://xenbits.xensource.com/linux-2.6.18-xen.hg
>
> Details for the solved issues:
> ======================
>
> - Whenever I boot my Xen 2.6.18.8 (Xen 3.3) with kernel parameter iommu=1
> the boot process freezes. (SOLVED)
>
> After the BIOS update the Xen kernel boots without problems even if I use the 
> iommu=1
>
> #----------------------------
>
> - VT-d broken. No PCI backend hiding with HVM guest possible (SOLVED)
>
> PCI backend hideing and VT-d seem to work now!
>
> The functionality 'pciback' is compiled fixed into my kernel (not a module).
>
> I use the following grub config
>
> title           Xen 3.3.0 / kernel 2.6.18.8-xen with IOMMU=1 and backhide 1G 
> NIC
> root            (hd0,0)
> kernel          /xen-3.3.0.gz console=vga noreboot max_loop=128 
> dom0_mem=1024M iommu=1
> module          /vmlinuz-2.6.18.8-xen root=/dev/mapper/RootVG-root ro 
> pciback.hide=(03:00.0)
> module          /initrd-2.6.18.8-xen.img
>
> The messages/dmesg show
>
> kernel: Bootdata ok (command line is root=/dev/mapper/RootVG-root ro 
> pciback.hide=(03:00.0))
> ...
> kernel: pciback 0000:03:00.0: seizing device
>
> The xm dmesg shows:
> (XEN) Xen version 3.3.0 (root@xxxxxxxxx) (gcc version 4.1.2 20061115 
> (prerelease) (Debian 4.1.1-21)) Tue Oct 21 16:22:47 CEST 2008
> (XEN) Latest ChangeSet: unavailable
> (XEN) Command line: console=vga noreboot max_loop=128 dom0_mem=1024M iommu=1
> (XEN) Video information:
> (XEN)  VGA is text mode 80x25, font 8x16
> (XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
> (XEN)  EDID info not retrieved because of reasons unknown
> (XEN) Disc information:
> (XEN)  Found 2 MBR signatures
> (XEN)  Found 2 EDD information structures
> (XEN) Xen-e820 RAM map:
> (XEN)  0000000000000000 - 0000000000096400 (usable)
> (XEN)  0000000000096400 - 00000000000a0000 (reserved)
> (XEN)  00000000000e0000 - 0000000000100000 (reserved)
> (XEN)  0000000000100000 - 00000000bfade000 (usable)
> (XEN)  00000000bfade000 - 00000000bfb21000 (ACPI NVS)
> (XEN)  00000000bfb21000 - 00000000bfc3f000 (reserved)
> (XEN)  00000000bfc3f000 - 00000000bfc42000 (ACPI NVS)
> (XEN)  00000000bfc42000 - 00000000bfd42000 (reserved)
> (XEN)  00000000bfd42000 - 00000000bfd43000 (ACPI NVS)
> (XEN)  00000000bfd43000 - 00000000bfd48000 (reserved)
> (XEN)  00000000bfd48000 - 00000000bfd50000 (ACPI data)
> (XEN)  00000000bfd50000 - 00000000bfd6b000 (ACPI NVS)
> (XEN)  00000000bfd6b000 - 00000000bfd8a000 (reserved)
> (XEN)  00000000bfd8a000 - 00000000bfd90000 (ACPI NVS)
> (XEN)  00000000bfd90000 - 00000000bff00000 (usable)
> (XEN)  00000000fed1c000 - 00000000fed20000 (reserved)
> (XEN)  00000000ff000000 - 0000000100000000 (reserved)
> (XEN)  0000000100000000 - 000000023c000000 (usable)
> (XEN) System RAM: 8123MB (8318864kB)
> (XEN) ACPI: RSDP 000F03C0, 0024 (r2  INTEL)
> (XEN) ACPI: XSDT BFD4EE18, 005C (r1 INTEL  DQ45CB         3D MSFT    10013)
> (XEN) ACPI: FACP BFD4DD98, 00F4 (r4  INTEL    A M I  6222004 MSFT    10013)
> (XEN) ACPI: DSDT BFD48018, 4E9A (r1 INTEL  DQ45CB         3D INTL 20051117)
> (XEN) ACPI: FACS BFD56F40, 0040
> (XEN) ACPI: APIC BFD4DF18, 006C (r2 INTEL  DQ45CB         3D MSFT    10013)
> (XEN) ACPI: MCFG BFD58E18, 003C (r1 INTEL  DQ45CB         3D MSFT       97)
> (XEN) ACPI: ASF! BFD57D18, 00A0 (r32 INTEL  DQ45CB         3D TFSM    F4240)
> (XEN) ACPI: SPCR BFD58D18, 0050 (r1 INTEL  DQ45CB         3D AMI.        3)
> (XEN) ACPI: TCPA BFD58C98, 0032 (r2 INTEL  DQ45CB         3D MSFT  1000013)
> (XEN) ACPI: DMAR BFD42F18, 00D8 (r1 INTEL  DQ45CB         3D INTL        1)
> (XEN) Xen heap: 14MB (14496kB)
> (XEN) Domain heap initialised
> (XEN) Processor #0 7:7 APIC version 20
> (XEN) Processor #1 7:7 APIC version 20
> (XEN) Processor #2 7:7 APIC version 20
> (XEN) Processor #3 7:7 APIC version 20
> (XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
> (XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
> (XEN) Intel VT-d has been enabled
> (XEN) Using scheduler: SMP Credit Scheduler (credit)
> (XEN) Detected 2833.076 MHz processor.
> (XEN) HVM: VMX enabled
> (XEN) CPU0: Intel(R) Core(TM)2 Quad  CPU   Q9550  @ 2.83GHz stepping 07
> (XEN) Booting processor 1/1 eip 8c000
> (XEN) CPU1: Intel(R) Core(TM)2 Quad  CPU   Q9550  @ 2.83GHz stepping 07
> (XEN) Booting processor 2/2 eip 8c000
> (XEN) CPU2: Intel(R) Core(TM)2 Quad  CPU   Q9550  @ 2.83GHz stepping 07
> (XEN) Booting processor 3/3 eip 8c000
> (XEN) CPU3: Intel(R) Core(TM)2 Quad  CPU   Q9550  @ 2.83GHz stepping 07
> (XEN) Total of 4 processors activated.
> (XEN) ENABLING IO-APIC IRQs
> (XEN)  -> Using new ACK method
> (XEN) checking TSC synchronization across 4 CPUs: passed.
> (XEN) Platform timer is 3.579MHz ACPI PM Timer
> (XEN) Brought up 4 CPUs
> (XEN) I/O virtualisation enabled
> (XEN) I/O virtualisation for PV guests disabled
> (XEN) mtrr: your CPUs had inconsistent fixed MTRR settings
> (XEN) *** LOADING DOMAIN 0 ***
> (XEN)  Xen  kernel: 64-bit, lsb, compat32
> (XEN)  Dom0 kernel: 64-bit, lsb, paddr 0xffffffff80200000 -> 
> 0xffffffff805c050c
> (XEN) PHYSICAL MEMORY ARRANGEMENT:
> (XEN)  Dom0 alloc.:   0000000230000000->0000000232000000 (253952 pages to be 
> allocated)
> (XEN) VIRTUAL MEMORY ARRANGEMENT:
> (XEN)  Loaded kernel: ffffffff80200000->ffffffff805c050c
> (XEN)  Init. ramdisk: ffffffff805c1000->ffffffff81506800
> (XEN)  Phys-Mach map: ffffffff81507000->ffffffff81707000
> (XEN)  Start info:    ffffffff81707000->ffffffff817074a4
> (XEN)  Page tables:   ffffffff81708000->ffffffff81717000
> (XEN)  Boot stack:    ffffffff81717000->ffffffff81718000
> (XEN)  TOTAL:         ffffffff80000000->ffffffff81800000
> (XEN)  ENTRY ADDRESS: ffffffff80200000
> (XEN) Dom0 has maximum 4 VCPUs
> (XEN) Scrubbing Free RAM: 
> .....................................................................done.
> (XEN) Xen trace buffers: disabled
> (XEN) Std. Loglevel: Errors and warnings
> (XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
> (XEN) Xen is relinquishing VGA console.
> (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to 
> Xen)
> (XEN) Freed 108kB init memory.
> (XEN) mtrr: type mismatch for c0000000,10000000 old: write-back new: 
> write-combining
>
> In the config of the HVM WinXP guest I use:
>
> pci=[ '03:00.0' ]
>
> In the running guest I see a new (native) 1Gbit NIC.
> I did not do any performance test yet.
> However the VT-d/PCI backend hide mechanism seems to work :-)
>
> #----------------------------
>
> - When I insert a PCIe x16 Graphics card (ATI Radeon HD 2400 Pro) the BIOS 
> menu display
> somehow becomes slow.  The experience is like if you use a 300 Baut serial 
> line connection.
> (SOLVED)
> Details see next paragraph
>
> - PCIe x16 graphics with Xen kernel broken
> SOLVED
>
> After the BIOS update I tested two PCIe x16 graphics adapters:
>  - ATI Radeon HD 2400 Pro
>  - MSI NX 8400GS (Nvidia 8400GS)
> Both cards run now without problems on a non-Xen standard Debian etch 
> 2.6.18.6 kernel.
> Especially, the BIOS menus are no longer slowed down.
> 3D runs perfectly with the propriatary drivers
>        ATI: ati-driver-installer-8-5-x86.x86_64.run
>        NVIDIA: NVIDIA-Linux-x86_64-177.80-pkg2.run
>
> Caveat: after testing the Nvida Card/Driver you need to remove some links 
> that point to
> NVIDIA drivers before you test again the ATI card/driver.
> Search /usr/lib for files like with '*177.80*' in them.
>
> With the xen-kernel 2.6.18.8 booted (details about version above) I was able 
> to configure the ATI card for X11
> by using the ATI driver ati-driver-installer-8-5-x86.x86_64.run.
> However, 3D s broken due to a problem with symbols
>
> Oct 27 10:09:09 susi kernel: fglrx: Unknown symbol xen_invlpg_mask
>
> With the xen-kernel 2.6.18.8 booted (details about version above) I was not 
> able to compile the NVIDIA driver
>  NVIDIA-Linux-x86_64-177.80-pkg2.run
> The build script from NVIDIA expclicitly states that it won't not run on a 
> XEN-kernel

The build script can be told to ignore Xen, I am running 177.80 on
2.6.27.3 (kernel.org + modified suse xen patches) with a single 9500GT
driving two 1920x1200 TFT's (not xinerama), I have a non xen kernel of
the same version installed so when I need to install nvidia drivers I
do the following:

rm /usr/src/linux
ln -s /usr/src/linux-2.6.27 /usr/src/linux
emerge nvidia-drivers #substitute the appropriate command for your distro
rm /usr/src/linux
ln -s /usr/src/linux-2.6.27-xen /usr/src/linux
cd /usr/src
sh /home/andy/Desktop/NVIDIA-Linux-x86_64-177.80-pkg2.run -x
cd NVIDIA-Linux-x86_64-177.80-pkg2/usr/src/nv/
IGNORE_XEN_PRESENCE=y SYSSRC=/usr/src/linux make module
IGNORE_XEN_PRESENCE=y SYSSRC=/usr/src/linux make install
modprobe nvidia

The drivers seem to work quite well under Xen, I can't really say they
are stable as my system occasionally locks up, but that happens
without the nvidia module/drivers loaded or X running at all and it
doesn't happen more frequently with them loaded so I think they are
probably ok.

Andy






>
> #-----------------------
>
> Details about BIOS update
>
> I download the archive
>        CBQ4510H.86A.0061.BI.ZIP
> from the Intel page
>
> I had quite some hard time to install the update via the DOS-Utility 
> IFLASH.EXE on a host without FD-drive
> and only linux installed on the HDs. After frantically searching for some old 
> Win boot media I came up with
> a good old Win98SE2 boot CD. It has the nice feature to recognize USB sticks 
> as FD disk drive B:
>
> Then I proceeded as follows.
> (Shame on me that I do not know how to do it better with purely Linux and 
> freeDos basis
>  Is anybody out there who can educate me?)
>
> Unpack the zip archive.
> This yields
>
> CB0061.BIO
> CB0061.ITK
> IFLASH.EXE
>
> Copy these files to a USB stick with FAT FS
>
> Reboot the host and enter BIOS
> Set Advanced/Drives Disk mode from RAID to IDE
> This allows for a proper Win98SE2 boot from CD
>
> Boot a Win98SE2 CD.
> It will recognize the USB stick as drive b:
>
> b:
> iflash /PF CB0061.BIO
>
> When prompted
> - remove CD
> - remove USB stick
>
> type return an wait until BIOS flash has finished
>
> It takes about 3-5 minutes
>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-users
>

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users