Re: PCI pass-through problem for SN570 NVME SSD

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Mon, Jul 04, 2022 at 10:51:53PM +0800, G.R. wrote:
> On Mon, Jul 4, 2022 at 9:09 PM Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
> > >
> > > 05:00.0 Non-Volatile memory controller: Sandisk Corp Device 501a (prog-if 
> > > 02 [NVM Express])
> > >       Subsystem: Sandisk Corp Device 501a
> > >       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> > > Stepping- SERR- FastB2B- DisINTx+
> > >       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> > > <TAbort- <MAbort- >SERR- <PERR- INTx-
> > >       Latency: 0, Cache Line Size: 64 bytes
> > >       Interrupt: pin A routed to IRQ 16
> > >       NUMA node: 0
> > >       IOMMU group: 13
> > >       Region 0: Memory at a2600000 (64-bit, non-prefetchable) [size=16K]
> > >       Region 4: Memory at a2604000 (64-bit, non-prefetchable) [size=256]
> >
> > I think I'm slightly confused, the overlapping happens at:
> >
> > (XEN) d1: GFN 0xf3078 (0xa2616,0,5,7) -> (0xa2504,0,5,7) not permitted
> >
> > So it's MFNs 0xa2616 and 0xa2504, yet none of those are in the BAR
> > ranges of this device.
> >
> > Can you paste the lspci -vvv output for any other device you are also
> > passing through to this guest?
> >
> 
> I just realized that the address may change in different environments.
> In previous email chains, I used a cached dump from a Linux
> environment running outside the hypervisor.
> Sorry for the confusion. Refreshing with a XEN dom0 dump.
> 
> The other device I used is a SATA controller. I think I can get what
> you are looking for now.
> Both a2616 and a2504 are found!
> 
> 00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI
> Controller (rev 10) (prog-if 01 [AHCI 1.0])
>         DeviceName: Onboard - SATA
>         Subsystem: Gigabyte Technology Co., Ltd Cannon Lake PCH SATA
> AHCI Controller
>         Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>         Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium
> >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Interrupt: pin A routed to IRQ 16
>         Region 0: Memory at a2610000 (32-bit, non-prefetchable) [size=8K]
>         Region 1: Memory at a2616000 (32-bit, non-prefetchable) [size=256]
>         Region 2: I/O ports at 4090 [size=8]
>         Region 3: I/O ports at 4080 [size=4]
>         Region 4: I/O ports at 4060 [size=32]
> 
> 05:00.0 Non-Volatile memory controller: Sandisk Corp Device 501a
> (prog-if 02 [NVM Express])
>         Subsystem: Sandisk Corp Device 501a
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0, Cache Line Size: 64 bytes
>         Interrupt: pin A routed to IRQ 11
>         Region 0: Memory at a2500000 (64-bit, non-prefetchable) [size=16K]
>         Region 4: Memory at a2504000 (64-bit, non-prefetchable) [size=256]

Right, so hvmloader attempts to place a BAR from 05:00.0 and a BAR
from 00:17.0 into the same page, which is not that good behavior.  It
might be sensible to attempt to share the page if both BARs belong to
the same device, but not if they belong to different devices.

I think the following patch:

https://lore.kernel.org/xen-devel/20200117110811.43321-1-roger.pau@xxxxxxxxxx/

Might help with this.

Thanks, Roger.