Re: [Xen-users] Debian Xen pciback missing?

[Casey DeLorme <cdelorme@xxxxxxxxx>]

Thanks for the follow-up Tobias, and for making today an incredible success!  I owe many of todays accomplishments to your initial suggestion:

-  Compiled Linux Kernel 3.2.9 (this was a first)

-  Verified pciback functioning from Dom0

-  Got a Remote VNC working

-  Installed a Windows 7 HVM DomU

-  Passed onboard HD Audio & USB 3.0

-  Successfully tested the onboard Audio (100% functional)

I began researching VT-d three months ago, and my mind was lost in a sea of innumerable possibilities.  I spent a month verifying compatible components, including selecting a motherboard.

ASUS and Gigabyte were ruled out due to there distinct lack of Socket 1155 boards with VT-d support.  I settled on ASRock as their price to feature was pretty decent, but only two 1155 boards had VT-d & Dual LAN, the Extreme7 and Fatal1ty.  I went with the Extreme7, because the Fatal1ty had useless (to me) components (IDE & PS/2).  Both boards have a PLX PEX8608 chip, so each would have at least one layer between the PCI card and the board, so I don't know if the Fatal1ty would solve my problem.

To clarify, the NF200 chip is built into the motherboard, it is not a part of the nVidia GPU, but it is made by nVidia.  Even if I had an ATI card it would still run through the same layer and be prone to the same problem.  The NF200 chip is used to manage SLI and CrossFire, so it isn't an nVidia feature exclusively.

I confirmed via lspci -vvv, the NF200 chip does not feature ACS functionality, hence the error.

I read that the security setting can be ignored by modifying the xend-config, but I won't be testing this until tomorrow.  Alternatively there is one PCIe slot not sitting behind the NF200 that I can also try.  In all likelihood though, I may end up swapping motherboards again.

I began working on this project 17 days ago, and since then I have read through roughly 3000~ pages of documentation.  I am on my 47th fresh installation of Linux, with over 40 pages of detailed notes.  Words cannot express how pleased I was to have audio today.

That said, if anyone has had any experience working with layered pci passthrough, or overriding the Xen security setting I would be most grateful to hear about it.

Signing off,

~Casey

On Wed, Mar 7, 2012 at 3:31 AM, Tobias Geiger <tobias.geiger@xxxxxxxxx> wrote:

Hi Casey,

looks like nvidia brings with it not only a GPU and a HDMI pci-device, but
also "NF200 PCIe 2.0 switch" ; Maybe this is one of the many reasons why
NVIDIA GPUs make so much problems when passing them through in xen.

Sorry, my short answer is: get a ATI Card :)
With ATI it's - generally - no problem to pass it through in xen. no such PCI-
Device-ID/BUS plethora, no xen code patches to make - it "just works", at
least here with a Radeon 6970.

You may test David Techer's way of adapting xen to your individual nvidia
setup, search (also in xen-devel) for his posts and urls to his blog.

Greetings and good luck anyways!
Tobias

Am Mittwoch, 7. März 2012, 01:55:05 schrieb Casey DeLorme:

> Hello again, and many thanks.  I bring a status update!
>
> I was able to compile the Linux Kernel 3.2.9 with pciback built-in instead
> of as a module.  I also installed Xen Hypervisor 4.1 via aptitude and was
> able to adjust settings in grub.cfg after.
>
> Xen boots, and the grub line appears to work:
> xen-pciback.hide=(01:00.0)(02:00.0)(03:00.0)(03:00.1)
>
> Now when I run xm pci-list-assignable-devices, I get:
> 0000:03:00.0
> 0000:03:00.1
> 0000:01:00.0
> 0000:02:00.0
> 0000:02:02.0
> 0000:02:03.0
>
> My objective is to achieve USB & VGA passthrough, but my graphics card
> appears to be sitting behind numerous devices.
>
> My DomU Configuration:
>
> name="windows7"
> maxmem=6912
> memory=6912
> #maxvcpus=6
> vcpus=4
> > > > > > > localtime=0
> builder="hvm"
> device_model="/usr/lib/xen-4.1/bin/qemu-dm"
> kernel="/usr/lib/xen-4.1/boot/hvmloader"
> boot="dc"
> disk=[
>     'phy:/dev/guests/win7,hda,w',
>     'file:/home/xen/win7.iso,hdc:cdrom,r'
>   ]
> vif=[ 'bridge=xenbr0, model=e1000' ]
> stdvga=1
> sdl=0
> vnc=1
> vncunused=1
> viridian=1
> pae=1
> usbdevice='tablet'
>
>
> I can run xm create -c windows7 without any problems.
>
> When I add this to the configuration:
>
> pci=[
>     '03:00.0',
>     '03:00.1'
>   ]
>
> I get this error:
> Error: pci: to avoid potential security issue, 0000:03:00.0 is not allowed
> to be assigned to guest since it is behind PCIe switch that does not
> support or enable ACS.
>
> So I tried this:
>
> pci=[
>     '01:00.0'
>     '03:00.0',
>     '03:00.1'
>   ]
>
> However, I then receive this error:
> Error: pci: PCI Backend and pci-stub don't own device 0000:01:00.0
>
> Even though it's listed as an assignable device, so I don't know if I
> misread what the card is sitting behind, or if there is some override I
> need to add to my configuration.
>
> I can pass other devices, such as my USB and onboard Audio without any
> problems, for example:
>
> pci=[
>     '00:1a.0'
>   ]
>
> If I try to attach the device after starting the DomU I get:
>
> xm pci-attach 3 0000:01:00.0
> Error: pci: PCI Backend and pci-stub don't own device 0000:01:00.0
> Usage: xm pci-attach [-o|--options=<opt>] <Domain> <domain:bus:slot.func>
> [virtual slot]
>
> Insert a new pass-through pci device.
>
>
> So, I have two questions.
>
> #1:  Is there a way to override the first error, if it's a "potential
> security" risk, surely there must be an option to ignore it?
>
> #2:  Any ideas as to why the 01:00.0 devices is listed as assignable yet
> not available?  Do I need to pass a step further up?
>
>
> lspci -tv output:
>
> -[0000:00]-+-00.0  Intel Corporation 2nd Generation Core Processor Family
> DRAM Controller
>            +-01.0-[01-05]----00.0-[02-05]--+-00.0-[03]--+-00.0  nVidia
> Corporation GF104 [GeForce GTX 460]
>
>            |                               |            \-00.1  nVidia
>
> Corporation GF104 High Definition Audio Controller
>
>            |                               +-02.0-[04]--
>            |                               \-03.0-[05]--
>
>            +-02.0  Intel Corporation 2nd Generation Core Processor Family
> Integrated Graphics Controller
>            +-16.0  Intel Corporation 6 Series/C200 Series Chipset Family
> MEI Controller #1
>            +-1a.0  Intel Corporation 6 Series/C200 Series Chipset Family
> USB Enhanced Host Controller #2
>            +-1b.0  Intel Corporation 6 Series/C200 Series Chipset Family
> High Definition Audio Controller
>            +-1c.0-[06]--
>            +-1c.5-[07]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA
> Controller
>            +-1c.6-[08]----00.0  ASMedia Technology Inc. ASM1042 SuperSpeed
> USB Host Controller
>            +-1c.7-[09-12]----00.0-[0a-12]--+-01.0-[0b]----00.0  Broadcom
> Corporation NetLink BCM57781 Gigabit Ethernet PCIe
>
>            |                               +-04.0-[0c]----00.0  ASMedia
>
> Technology Inc. ASM1062 Serial ATA Controller
>
>            |                               +-05.0-[0d]----00.0  Broadcom
>
> Corporation NetLink BCM57781 Gigabit Ethernet PCIe
>
>            |                               +-06.0-[0e]----00.0  VIA
>
> Technologies, Inc. VT6315 Series Firewire Controller
>
>            |                               +-07.0-[0f-10]----00.0-[10]--
>            |                               +-08.0-[11]----00.0  ASMedia
>
> Technology Inc. ASM1042 SuperSpeed USB Host Controller
>
>            |                               \-09.0-[12]----00.0  ASMedia
>
> Technology Inc. ASM1042 SuperSpeed USB Host Controller
>            +-1d.0  Intel Corporation 6 Series/C200 Series Chipset Family
> USB Enhanced Host Controller #1
>            +-1f.0  Intel Corporation Z68 Express Chipset Family LPC
> Controller
>            +-1f.2  Intel Corporation 6 Series/C200 Series Chipset Family 6
> port SATA AHCI Controller
>            \-1f.3  Intel Corporation 6 Series/C200 Series Chipset Family
> SMBus Controller
>
> lspci:
>
> 00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family
> DRAM Controller (rev 09)
> 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core
> Processor Family PCI Express Root Port (rev 09)
> 00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core
> Processor Family Integrated Graphics Controller (rev 09)
> 00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series
> Chipset Family MEI Controller #1 (rev 04)
> 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
> Family USB Enhanced Host Controller #2 (rev 05)
> 00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family
> High Definition Audio Controller (rev 05)
> 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family
> PCI Express Root Port 1 (rev b5)
> 00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family
> PCI Express Root Port 6 (rev b5)
> 00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family
> PCI Express Root Port 7 (rev b5)
> 00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family
> PCI Express Root Port 8 (rev b5)
> 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
> Family USB Enhanced Host Controller #1 (rev 05)
> 00:1f.0 ISA bridge: Intel Corporation Z68 Express Chipset Family LPC
> Controller (rev 05)
> 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset
> Family 6 port SATA AHCI Controller (rev 05)
> 00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus
> Controller (rev 05)
> 01:00.0 PCI bridge: nVidia Corporation NF200 PCIe 2.0 switch for mainboards
> (rev a3)
> 02:00.0 PCI bridge: nVidia Corporation NF200 PCIe 2.0 switch for mainboards
> (rev a3)
> 02:02.0 PCI bridge: nVidia Corporation NF200 PCIe 2.0 switch for mainboards
> (rev a3)
> 02:03.0 PCI bridge: nVidia Corporation NF200 PCIe 2.0 switch for mainboards
> (rev a3)
> 03:00.0 VGA compatible controller: nVidia Corporation GF104 [GeForce GTX
> 460] (rev a1)
> 03:00.1 Audio device: nVidia Corporation GF104 High Definition Audio
> Controller (rev a1)
> 07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA
> Controller (rev 01)
> 08:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host
> Controller
> 09:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
> Express Gen 2 (5.0 GT/s) Switch (rev ba)
> 0a:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
> Express Gen 2 (5.0 GT/s) Switch (rev ba)
> 0a:04.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
> Express Gen 2 (5.0 GT/s) Switch (rev ba)
> 0a:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
> Express Gen 2 (5.0 GT/s) Switch (rev ba)
> 0a:06.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
> Express Gen 2 (5.0 GT/s) Switch (rev ba)
> 0a:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
> Express Gen 2 (5.0 GT/s) Switch (rev ba)
> 0a:08.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
> Express Gen 2 (5.0 GT/s) Switch (rev ba)
> 0a:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
> Express Gen 2 (5.0 GT/s) Switch (rev ba)
> 0b:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit
> Ethernet PCIe (rev 10)
> 0c:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA
> Controller (rev 01)
> 0d:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit
> Ethernet PCIe (rev 10)
> 0e:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6315 Series Firewire
> Controller (rev 01)
> 0f:00.0 PCI bridge: ASMedia Technology Inc. ASM108x PCIe to PCI Bridge
> Controller (rev 01)
> 11:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host
> Controller
> 12:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host
> Controller
>
> On Tue, Mar 6, 2012 at 1:29 PM, Casey DeLorme <cdelorme@xxxxxxxxx> wrote:
> > Hello Tobias, and thanks for the suggestions!
> >
> > I had been told that Debian Squeeze came with pciback "out of the box",
> > but clearly that was a misnomer (given the "m" in my kernel config), and
> > as mentioned in my initial email I tried adding lines to grub with no
> > results.
> >
> > Using the find command I was able to find three records with pci:
> > /lib/modules/3.2.0-1-amd64/kernel/drivers/pci/xen-pcifront.ko
> > /lib/modules/3.2.0-1-amd64/kernel/drivers/xen/xen-pciback
> > /lib/modules/3.2.0-1-amd64/kernel/drivers/xen/xen-pciback/xen-pciback.ko
> >
> > Also, I can type just "modprobe xen-pciback" without any errors.
> > However, when I attempt to send a command to it, it fails:
> > sudo modprobe xen-pciback 'hide=(03:00.0)'
> > ERROR: could not insert 'xen_pciback': No such file or directory
> >
> > I am new enough to Xen that this really doesn't give me enough to go on.
> >
> >  It's trying to insert... someplace, and it's using xen_pciback (not
> >
> > xen-pciback?)
> >
> >
> > Your other suggestion is much more appealing, but I want to ask a few
> > questions before I give that a spin.
> >
> > Question #1:
> > I would be compiling the whole kernel, correct?  I just want to make sure
> > I'm not missing some faster or easier method that would append
> > functionality to an existing kernel.
> >
> > Question #2:
> > With a custom Kernel will I be able to use the package manager to install
> > Xen, or will I have to compile Xen from source to have it recognize my
> > custom Linux Kernel?
> >
> > Question #3:
> > If I wanted to compile Xen from source, or the package manager is not an
> > option, how can I compile the latest Xen source with my custom kernel?
> >
> > I found instructions for 4.1.1 in a pdf, but they didn't work verbatim
> > with 4.1.2, and I ended up with 4x .gz files in my boot directory, and
> > half a dozen new grub records, none of which booted.
> >
> > Compiled from source would be ideal as I could try fixing the efi-boot
> > memory issues, but without some kind of guide for 4.1.2 I would be lost
> > at accomplishing this.
> >
> > ~Casey

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users