Re: [Xen-devel] Xen 4.3 PCI passthrough possible bug

[Mike Neiderhauser <mikeneiderhauser@xxxxxxxxx>]

So I went ahead to tested the setup I am trying to achieve using xen.  This setup basically requires two isolated machines that can be used for network testing.  On the hvm mentioned above, this testing fails due to something I cannot wrap my head around.  I believe it is still related to the PCI passthrough of a device and I believe it is related to the libxl error mentioned above. Can anyone shed some light on what is going on?  Is it a driver issue? (Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet)

[79464.816085] ------------[ cut here ]------------

[79464.816093] WARNING: at /build/buildd/linux-lts-raring-3.8.0/net/sched/sch_generic.c:254 dev_watchdog+0x262/0x270()

[79464.816094] Hardware name: HVM domU

[79464.816096] NETDEV WATCHDOG: eth1 (bnx2): transmit queue 1 timed out

[79464.816096] Modules linked in: esp6(F) ah6(F) xfrm6_mode_tunnel(F) xfrm_user(F) xfrm4_tunnel(F) tunnel4(F) ipcomp(F) xfrm_ipcomp(F) authenc(F) esp4(F) ah4(F) xfrm4_mode_tunnel(F) deflate(F) zlib_deflate(F) ctr(F) twofish_generic(F) twofish_avx_x86_64(F) twofish_x86_64_3way(F) twofish_x86_64(F) twofish_common(F) camellia_generic(F) camellia_aesni_avx_x86_64(F) camellia_x86_64(F) serpent_avx_x86_64(F) serpent_sse2_x86_64(F) glue_helper(F) serpent_generic(F) blowfish_generic(F) blowfish_x86_64(F) blowfish_common(F) cast5_avx_x86_64(F) cast5_generic(F) cast_common(F) des_generic(F) xcbc(F) rmd160(F) crypto_null(F) af_key(F) xfrm_algo(F) 8021q(F) garp(F) stp(F) llc(F) ipmi_msghandler(F) autofs4(F) ghash_clmulni_intel(F) aesni_intel(F) ablk_helper(F) cryptd(F) lrw(F) aes_x86_64(F) xts(F) gf128mul(F) cirrus(F) ttm(F) drm_kms_helper(F) nfsd(F) nfs_acl(F) drm(F) auth_rpcgss(F) sysimgblt(F) nfs(F) psmouse(F) sysfillrect(F) syscopyarea(F) fscache(F) lockd(F) microcode(F) joydev(F) xen_kbdfront(F) i2c_piix4(F) serio_raw(F) lp(F) mac_hid(F) sunrpc(F) parport(F) floppy(F) bnx2(F) [last unloaded: ipmi_msghandler]

[79464.816139] Pid: 0, comm: swapper/1 Tainted: GF            3.8.0-29-generic #42~precise1-Ubuntu

[79464.816140] Call Trace:

[79464.816142]  <IRQ>  [<ffffffff81059b0f>] warn_slowpath_common+0x7f/0xc0

[79464.816149]  [<ffffffff8135b9d4>] ? timerqueue_add+0x64/0xb0

[79464.816151]  [<ffffffff81059c06>] warn_slowpath_fmt+0x46/0x50

[79464.816154]  [<ffffffff81076794>] ? wake_up_worker+0x24/0x30

[79464.816157]  [<ffffffff81602062>] dev_watchdog+0x262/0x270

[79464.816160]  [<ffffffff810771f0>] ? __queue_work+0x2d0/0x2d0

[79464.816161]  [<ffffffff81601e00>] ? pfifo_fast_dequeue+0xe0/0xe0

[79464.816164]  [<ffffffff8106995b>] call_timer_fn+0x3b/0x150

[79464.816167]  [<ffffffff8144f5a1>] ? add_interrupt_randomness+0x41/0x190

[79464.816170]  [<ffffffff8106b427>] run_timer_softirq+0x267/0x2c0

[79464.816173]  [<ffffffff810ee3c9>] ? handle_irq_event_percpu+0xa9/0x210

[79464.816175]  [<ffffffff81601e00>] ? pfifo_fast_dequeue+0xe0/0xe0

[79464.816177]  [<ffffffff81062620>] __do_softirq+0xc0/0x240

[79464.816180]  [<ffffffff810b67cd>] ? tick_do_update_jiffies64+0x9d/0xd0

[79464.816184]  [<ffffffff816fdd5c>] call_softirq+0x1c/0x30

[79464.816188]  [<ffffffff81016775>] do_softirq+0x65/0xa0

[79464.816189]  [<ffffffff810628fe>] irq_exit+0x8e/0xb0

[79464.816193]  [<ffffffff8140a125>] xen_evtchn_do_upcall+0x35/0x50

[79464.816195]  [<ffffffff816fdeed>] xen_hvm_callback_vector+0x6d/0x80

[79464.816196]  <EOI>  [<ffffffff81084008>] ? hrtimer_start+0x18/0x20

[79464.816201]  [<ffffffff81045136>] ? native_safe_halt+0x6/0x10

[79464.816204]  [<ffffffff8101cc33>] default_idle+0x53/0x1f0

[79464.816206]  [<ffffffff8101dad9>] cpu_idle+0xd9/0x120

[79464.816209]  [<ffffffff816d10fe>] start_secondary+0xc3/0xc5

[79464.816210] ---[ end trace 48cf6b13be16e0ae ]---

[79464.816214] bnx2 0000:00:05.0 eth1: <--- start FTQ dump --->

[79464.816220] bnx2 0000:00:05.0 eth1: RV2P_PFTQ_CTL 00010000

[79464.816225] bnx2 0000:00:05.0 eth1: RV2P_TFTQ_CTL 00020000

[79464.816258] bnx2 0000:00:05.0 eth1: RV2P_MFTQ_CTL 00004000

[79464.816263] bnx2 0000:00:05.0 eth1: TBDR_FTQ_CTL 00004000

[79464.816268] bnx2 0000:00:05.0 eth1: TDMA_FTQ_CTL 00010000

[79464.816272] bnx2 0000:00:05.0 eth1: TXP_FTQ_CTL 00010000

[79464.816276] bnx2 0000:00:05.0 eth1: TXP_FTQ_CTL 00010000

[79464.816280] bnx2 0000:00:05.0 eth1: TPAT_FTQ_CTL 00010000

[79464.816285] bnx2 0000:00:05.0 eth1: RXP_CFTQ_CTL 00008000

[79464.816289] bnx2 0000:00:05.0 eth1: RXP_FTQ_CTL 00100000

[79464.816293] bnx2 0000:00:05.0 eth1: COM_COMXQ_FTQ_CTL 00010000

[79464.816297] bnx2 0000:00:05.0 eth1: COM_COMTQ_FTQ_CTL 00020000

[79464.816302] bnx2 0000:00:05.0 eth1: COM_COMQ_FTQ_CTL 00010000

[79464.816306] bnx2 0000:00:05.0 eth1: CP_CPQ_FTQ_CTL 00004000

[79464.816308] bnx2 0000:00:05.0 eth1: CPU states:

[79464.816321] bnx2 0000:00:05.0 eth1: 045000 mode b84c state 80001000 evt_mask 500 pc 8001288 pc 8001288 instr 38640001

[79464.816335] bnx2 0000:00:05.0 eth1: 085000 mode b84c state 80005000 evt_mask 500 pc 8000a5c pc 8000a5c instr 10400016

[79464.816349] bnx2 0000:00:05.0 eth1: 0c5000 mode b84c state 80000000 evt_mask 500 pc 8004c14 pc 8004c14 instr 32050003

[79464.816362] bnx2 0000:00:05.0 eth1: 105000 mode b8cc state 80000000 evt_mask 500 pc 8000a9c pc 8000a94 instr 8c420020

[79464.816375] bnx2 0000:00:05.0 eth1: 145000 mode b880 state 80000000 evt_mask 500 pc 8009c00 pc 800d948 instr 30420040

[79464.816389] bnx2 0000:00:05.0 eth1: 185000 mode b8cc state 80008000 evt_mask 500 pc 8000c58 pc 8000c58 instr 1092823

[79464.816392] bnx2 0000:00:05.0 eth1: <--- end FTQ dump --->

[79464.816394] bnx2 0000:00:05.0 eth1: <--- start TBDC dump --->

[79464.816399] bnx2 0000:00:05.0 eth1: TBDC free cnt: 32

[79464.816401] bnx2 0000:00:05.0 eth1: LINE     CID  BIDX   CMD  VALIDS

[79464.816410] bnx2 0000:00:05.0 eth1: 00    001000  00e8   00    [0]

[79464.816420] bnx2 0000:00:05.0 eth1: 01    001000  00e8   00    [0]

[79464.816429] bnx2 0000:00:05.0 eth1: 02    000800  afc8   00    [0]

[79464.816438] bnx2 0000:00:05.0 eth1: 03    000800  afb8   00    [0]

[79464.816447] bnx2 0000:00:05.0 eth1: 04    000800  afd8   00    [0]

[79464.816456] bnx2 0000:00:05.0 eth1: 05    000800  afe0   00    [0]

[79464.816465] bnx2 0000:00:05.0 eth1: 06    000800  afe8   00    [0]

[79464.816474] bnx2 0000:00:05.0 eth1: 07    000800  afd0   00    [0]

[79464.816485] bnx2 0000:00:05.0 eth1: 08    001000  3510   00    [0]

[79464.816494] bnx2 0000:00:05.0 eth1: 09    000800  aec0   00    [0]

[79464.816504] bnx2 0000:00:05.0 eth1: 0a    001000  3530   00    [0]

[79464.816514] bnx2 0000:00:05.0 eth1: 0b    000800  aec8   00    [0]

[79464.816523] bnx2 0000:00:05.0 eth1: 0c    000800  aed0   00    [0]

[79464.816559] bnx2 0000:00:05.0 eth1: 0d    001000  34f8   00    [0]

[79464.816570] bnx2 0000:00:05.0 eth1: 0e    001000  3500   00    [0]

[79464.816580] bnx2 0000:00:05.0 eth1: 0f    001000  3518   00    [0]

[79464.816590] bnx2 0000:00:05.0 eth1: 10    1fbc00  2fe8   7d    [0]

[79464.816599] bnx2 0000:00:05.0 eth1: 11    1ab780  fff8   7d    [0]

[79464.816608] bnx2 0000:00:05.0 eth1: 12    17ff00  b908   f7    [0]

[79464.816618] bnx2 0000:00:05.0 eth1: 13    0cb700  ff40   d7    [0]

[79464.816627] bnx2 0000:00:05.0 eth1: 14    177a80  efe0   03    [0]

[79464.816637] bnx2 0000:00:05.0 eth1: 15    037d80  9f88   72    [0]

[79464.816646] bnx2 0000:00:05.0 eth1: 16    1bae00  eef8   ce    [0]

[79464.816657] bnx2 0000:00:05.0 eth1: 17    1bbc80  a7f8   df    [0]

[79464.816666] bnx2 0000:00:05.0 eth1: 18    17e180  6aa8   e4    [0]

[79464.816675] bnx2 0000:00:05.0 eth1: 19    07ff80  6e50   dd    [0]

[79464.816683] bnx2 0000:00:05.0 eth1: 1a    1fda80  f790   6e    [0]

[79464.816694] bnx2 0000:00:05.0 eth1: 1b    151580  d7b0   fc    [0]

[79464.816703] bnx2 0000:00:05.0 eth1: 1c    1b9f80  cef8   6b    [0]

[79464.816712] bnx2 0000:00:05.0 eth1: 1d    1ebf00  ffa8   df    [0]

[79464.816723] bnx2 0000:00:05.0 eth1: 1e    1e7e00  ff78   ef    [0]

[79464.816731] bnx2 0000:00:05.0 eth1: 1f    166e80  fbd8   aa    [0]

[79464.816733] bnx2 0000:00:05.0 eth1: <--- end TBDC dump --->

[79464.817101] bnx2 0000:00:05.0 eth1: DEBUG: intr_sem[0] PCI_CMD[00100406]

[79464.817239] bnx2 0000:00:05.0 eth1: DEBUG: PCI_PM[19002008] PCI_MISC_CFG[92000088]

[79464.817248] bnx2 0000:00:05.0 eth1: DEBUG: EMAC_TX_STATUS[00000008] EMAC_RX_STATUS[00000000]

[79464.817252] bnx2 0000:00:05.0 eth1: DEBUG: RPM_MGMT_PKT_CTRL[40000088]

[79464.817257] bnx2 0000:00:05.0 eth1: DEBUG: HC_STATS_INTERRUPT_STATUS[01fc0003]

[79464.817261] bnx2 0000:00:05.0 eth1: DEBUG: PBA[00000003]

[79464.817264] bnx2 0000:00:05.0 eth1: <--- start MCP states dump --->

[79464.817270] bnx2 0000:00:05.0 eth1: DEBUG: MCP_STATE_P0[0003611e] MCP_STATE_P1[0003611e]

[79464.817278] bnx2 0000:00:05.0 eth1: DEBUG: MCP mode[0000b880] state[80000000] evt_mask[00000500]

[79464.817286] bnx2 0000:00:05.0 eth1: DEBUG: pc[08003ebc] pc[0800d994] instr[32020020]

[79464.817288] bnx2 0000:00:05.0 eth1: DEBUG: shmem states:

[79464.817296] bnx2 0000:00:05.0 eth1: DEBUG: drv_mb[01030027] fw_mb[00000027] link_status[0008506b]

[79464.817301]  drv_pulse_mb[0000338a]

[79464.817306] bnx2 0000:00:05.0 eth1: DEBUG: dev_info_signature[44564907] reset_type[01005254]

[79464.817310]  condition[0003611e]

[79464.817318] bnx2 0000:00:05.0 eth1: DEBUG: 000001c0: 01005254 42530083 0003611e 00000000

[79464.817328] bnx2 0000:00:05.0 eth1: DEBUG: 000003cc: 00000000 00000000 00000000 00000000

[79464.817357] bnx2 0000:00:05.0 eth1: DEBUG: 000003dc: 00000000 00000000 00000000 00000000

[79464.817367] bnx2 0000:00:05.0 eth1: DEBUG: 000003ec: 00000000 00000000 00000000 00000000

[79464.817372] bnx2 0000:00:05.0 eth1: DEBUG: 0x3fc[00000000]

[79464.817374] bnx2 0000:00:05.0 eth1: <--- end MCP states dump --->

[79464.872988] bnx2 0000:00:05.0 eth1: NIC Copper Link is Down

On Mon, Feb 10, 2014 at 1:47 PM, Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx> wrote:

On Mon, Feb 10, 2014 at 01:29:22PM -0500, Mike Neiderhauser wrote:
> Thanks for the answers on the timeline.
>
> When I start the HVM with th broadcom adapter, I get this message back.
> Parsing config from /etc/xen/ubuntu-hvm-1.cfg
> libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't
> support reset from sysfs for PCI device 0000:05:00.0
> libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't
> support reset from sysfs for PCI device 0000:05:00.1
>
> However, the devices appear in the HVM.  Is this something that I should be
> concerned about?

No. Xen pciback does the reset automatically.

Actually we might want to ditch that reporting in libxl, or maybe just
implement a stub function in xen-pciback so that libxl will be happy.

>
>
>
> On Mon, Feb 10, 2014 at 12:11 PM, George Dunlap <dunlapg@xxxxxxxxx> wrote:
>
> > On Sat, Feb 8, 2014 at 5:42 PM, Mike Neiderhauser
> > <mikeneiderhauser@xxxxxxxxx> wrote:
> > > Works like a charm.  I do not have physical access to the computer this
> > > weekend to verify that the cards are isolated, but the HVM starts and
> > > appears to be working well.
> > >
> > > When do you think Xen 4.4 will be released.  The article I read
> > mentioned it
> > > will be released in 2014 (hinting towards the end of February).  I also
> > read
> > > 'When it is ready.'
> > >
> > > Any timeline would be great.
> >
> > I'm afraid that's about all we can give. :-)  We've locked down
> > development for 2 months now and are working on finding and fixing
> > bugs.  If there are no more blocker bugs or other unforeseen delays,
> > it should be out by the end of February.  But there are necessarily
> > significant unknowns, so we can't make any promises.
> >
> >  -George
> >

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel