Re: PCI pass-through problem for SN570 NVME SSD

["G.R." <firemeteor@xxxxxxxxxxxxxxxxxxxxx>]

On Mon, Jul 4, 2022 at 10:51 PM G.R. <firemeteor@xxxxxxxxxxxxxxxxxxxxx> wrote:
>
> On Mon, Jul 4, 2022 at 9:09 PM Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
> > >
> > > 05:00.0 Non-Volatile memory controller: Sandisk Corp Device 501a (prog-if 
> > > 02 [NVM Express])
> > >       Subsystem: Sandisk Corp Device 501a
> > >       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> > > Stepping- SERR- FastB2B- DisINTx+
> > >       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> > > <TAbort- <MAbort- >SERR- <PERR- INTx-
> > >       Latency: 0, Cache Line Size: 64 bytes
> > >       Interrupt: pin A routed to IRQ 16
> > >       NUMA node: 0
> > >       IOMMU group: 13
> > >       Region 0: Memory at a2600000 (64-bit, non-prefetchable) [size=16K]
> > >       Region 4: Memory at a2604000 (64-bit, non-prefetchable) [size=256]
> >
> > I think I'm slightly confused, the overlapping happens at:
> >
> > (XEN) d1: GFN 0xf3078 (0xa2616,0,5,7) -> (0xa2504,0,5,7) not permitted
> >
> > So it's MFNs 0xa2616 and 0xa2504, yet none of those are in the BAR
> > ranges of this device.
> >
> > Can you paste the lspci -vvv output for any other device you are also
> > passing through to this guest?
> >

As reminded by this request, I tried to assign this nvme device to
another FreeBSD12 domU.
This time it does not fail at the VM setup stage, but the device is
still not usable at the domU.
The nvmecontrol command is not able to talk to the device at all:
nvme0: IDENTIFY (06) sqid:0 cid:0 nsid:0 cdw10:00000001 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) sqid:0 cid:0 cdw0:0
nvme0: IDENTIFY (06) sqid:0 cid:0 nsid:0 cdw10:00000001 cdw11:00000000
nvme0: ABORTED - BY REQUEST (00/07) sqid:0 cid:0 cdw0:0

The QEMU log says the following:
00:05.0] Write-back to unknown field 0x09 (partially) inhibited (0x00)
[00:05.0] If the device doesn't work, try enabling permissive mode
[00:05.0] (unsafe) and if it helps report the problem to xen-devel
[00:05.0] msi_msix_setup: Error: Mapping of MSI-X (err: 61, vec: 0x30, entry 0)

xl dmesg says the following:
(XEN) d[IO]: assign (0000:05:00.0) failed (-16)
(XEN) HVM d5v0 save: CPU
(XEN) HVM d5v1 save: CPU
(XEN) HVM d5v2 save: CPU
(XEN) HVM d5v3 save: CPU
(XEN) HVM d5 save: PIC
(XEN) HVM d5 save: IOAPIC
(XEN) HVM d5v0 save: LAPIC
(XEN) HVM d5v1 save: LAPIC
(XEN) HVM d5v2 save: LAPIC
(XEN) HVM d5v3 save: LAPIC
(XEN) HVM d5v0 save: LAPIC_REGS
(XEN) HVM d5v1 save: LAPIC_REGS
(XEN) HVM d5v2 save: LAPIC_REGS
(XEN) HVM d5v3 save: LAPIC_REGS
(XEN) HVM d5 save: PCI_IRQ
(XEN) HVM d5 save: ISA_IRQ
(XEN) HVM d5 save: PCI_LINK
(XEN) HVM d5 save: PIT
(XEN) HVM d5 save: RTC
(XEN) HVM d5 save: HPET
(XEN) HVM d5 save: PMTIMER
(XEN) HVM d5v0 save: MTRR
(XEN) HVM d5v1 save: MTRR
(XEN) HVM d5v2 save: MTRR
(XEN) HVM d5v3 save: MTRR
(XEN) HVM d5 save: VIRIDIAN_DOMAIN
(XEN) HVM d5v0 save: CPU_XSAVE
(XEN) HVM d5v1 save: CPU_XSAVE
(XEN) HVM d5v2 save: CPU_XSAVE
(XEN) HVM d5v3 save: CPU_XSAVE
(XEN) HVM d5v0 save: VIRIDIAN_VCPU
(XEN) HVM d5v1 save: VIRIDIAN_VCPU
(XEN) HVM d5v2 save: VIRIDIAN_VCPU
(XEN) HVM d5v3 save: VIRIDIAN_VCPU
(XEN) HVM d5v0 save: VMCE_VCPU
(XEN) HVM d5v1 save: VMCE_VCPU
(XEN) HVM d5v2 save: VMCE_VCPU
(XEN) HVM d5v3 save: VMCE_VCPU
(XEN) HVM d5v0 save: TSC_ADJUST
(XEN) HVM d5v1 save: TSC_ADJUST
(XEN) HVM d5v2 save: TSC_ADJUST
(XEN) HVM d5v3 save: TSC_ADJUST
(XEN) HVM d5v0 save: CPU_MSR
(XEN) HVM d5v1 save: CPU_MSR
(XEN) HVM d5v2 save: CPU_MSR
(XEN) HVM d5v3 save: CPU_MSR
(XEN) HVM5 restore: CPU 0
(XEN) d5: bind: m_gsi=16 g_gsi=36 dev=00.00.5 intx=0
(d5) HVM Loader
(d5) Detected Xen v4.14.3
(d5) Xenbus rings @0xfeffc000, event channel 1
(d5) System requested SeaBIOS
(d5) CPU speed is 3505 MHz
(d5) Relocating guest memory for lowmem MMIO space disabled
(d5) PCI-ISA link 0 routed to IRQ5
(d5) PCI-ISA link 1 routed to IRQ10
(d5) PCI-ISA link 2 routed to IRQ11
(d5) PCI-ISA link 3 routed to IRQ5
(d5) pci dev 01:3 INTA->IRQ10
(d5) pci dev 02:0 INTA->IRQ11
(d5) pci dev 04:0 INTA->IRQ5
(d5) pci dev 05:0 INTA->IRQ10
(d5) No RAM in high memory; setting high_mem resource base to 100000000
(d5) pci dev 03:0 bar 10 size 002000000: 0f0000008
(d5) pci dev 02:0 bar 14 size 001000000: 0f2000008
(d5) pci dev 04:0 bar 30 size 000040000: 0f3000000
(d5) pci dev 04:0 bar 10 size 000020000: 0f3040000
(d5) pci dev 03:0 bar 30 size 000010000: 0f3060000
(d5) pci dev 05:0 bar 10 size 000004000: 0f3070004
(d5) pci dev 03:0 bar 14 size 000001000: 0f3074000
(d5) pci dev 02:0 bar 10 size 000000100: 00000c001
(d5) pci dev 05:0 bar 20 size 000000100: 0f3075004
(d5) pci dev 04:0 bar 14 size 000000040: 00000c101
(d5) pci dev 01:1 bar 20 size 000000010: 00000c141
(XEN) memory_map:add: dom5 gfn=f3070 mfn=a2500 nr=2
(XEN) memory_map:add: dom5 gfn=f3073 mfn=a2503 nr=1
(XEN) memory_map:add: dom5 gfn=f3075 mfn=a2504 nr=1
(d5) Multiprocessor initialisation:
(d5)  - CPU0 ... 39-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(d5)  - CPU1 ... 39-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(d5)  - CPU2 ... 39-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(d5)  - CPU3 ... 39-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(d5) Writing SMBIOS tables ...
(d5) Loading SeaBIOS ...
(d5) Creating MP tables ...
(d5) Loading ACPI ...
(d5) vm86 TSS at fc100300
(d5) BIOS map:
(d5)  10000-100e3: Scratch space
(d5)  c0000-fffff: Main BIOS
(d5) E820 table:
(d5)  [00]: 00000000:00000000 - 00000000:000a0000: RAM
(d5)  HOLE: 00000000:000a0000 - 00000000:000c0000
(d5)  [01]: 00000000:000c0000 - 00000000:00100000: RESERVED
(d5)  [02]: 00000000:00100000 - 00000000:7f800000: RAM
(d5)  HOLE: 00000000:7f800000 - 00000000:fc000000
(d5)  [03]: 00000000:fc000000 - 00000000:fc00b000: NVS
(d5)  [04]: 00000000:fc00b000 - 00000001:00000000: RESERVED
(d5) Invoking SeaBIOS ...
(d5) SeaBIOS (version rel-1.13.0-1-gd542924-Xen)
(d5) BUILD: gcc: (Debian 11.2.0-13) 11.2.0 binutils: (GNU Binutils for
Debian) 2.37
(d5)
(d5) Found Xen hypervisor signature at 40000000
(XEN) memory_map:remove: dom5 gfn=f3070 mfn=a2500 nr=2
(XEN) memory_map:remove: dom5 gfn=f3073 mfn=a2503 nr=1
(XEN) memory_map:remove: dom5 gfn=f3075 mfn=a2504 nr=1
(XEN) memory_map:add: dom5 gfn=f3070 mfn=a2500 nr=2
(XEN) memory_map:add: dom5 gfn=f3073 mfn=a2503 nr=1
(XEN) memory_map:add: dom5 gfn=f3075 mfn=a2504 nr=1
(XEN) memory_map:remove: dom5 gfn=f3070 mfn=a2500 nr=2
(XEN) memory_map:remove: dom5 gfn=f3073 mfn=a2503 nr=1
(XEN) memory_map:remove: dom5 gfn=f3075 mfn=a2504 nr=1
(XEN) memory_map:add: dom5 gfn=f3070 mfn=a2500 nr=2
(XEN) memory_map:add: dom5 gfn=f3073 mfn=a2503 nr=1
(XEN) memory_map:add: dom5 gfn=f3075 mfn=a2504 nr=1
(XEN) memory_map:remove: dom5 gfn=f3070 mfn=a2500 nr=2
(XEN) memory_map:remove: dom5 gfn=f3073 mfn=a2503 nr=1
(XEN) memory_map:remove: dom5 gfn=f3075 mfn=a2504 nr=1
(XEN) memory_map:add: dom5 gfn=f3070 mfn=a2500 nr=2
(XEN) memory_map:add: dom5 gfn=f3073 mfn=a2503 nr=1
(XEN) memory_map:add: dom5 gfn=f3075 mfn=a2504 nr=1
(XEN) memory_map:remove: dom5 gfn=f3070 mfn=a2500 nr=2
(XEN) memory_map:remove: dom5 gfn=f3073 mfn=a2503 nr=1
(XEN) memory_map:remove: dom5 gfn=f3075 mfn=a2504 nr=1
(XEN) memory_map:add: dom5 gfn=f3070 mfn=a2500 nr=2
(XEN) memory_map:add: dom5 gfn=f3073 mfn=a2503 nr=1
(XEN) memory_map:add: dom5 gfn=f3075 mfn=a2504 nr=1