RE: Linux pin_user_pages_fast fails on Xen

["NK, JESHWANTHKUMAR (JESHWANTH KUMAR)" <JESHWANTHKUMAR.NK@xxxxxxx>]

[AMD Official Use Only - General]

Hi Stefano,

Thanks for the suggestion,

>Do you know if it works if you remove VM_IO | VM_PFNMAP from privcmd_mmap?
>> Gave a try, looks like the DomU doesn't boot without these two flags.

Regards,
Jeshwanth

-----Original Message-----
From: Stefano Stabellini <stefano.stabellini@xxxxxxx> 
Sent: Wednesday, September 14, 2022 5:01 AM
To: NK, JESHWANTHKUMAR (JESHWANTH KUMAR) <JESHWANTHKUMAR.NK@xxxxxxx>
Cc: Stabellini, Stefano <stefano.stabellini@xxxxxxx>; 
boris.ostrovsky@xxxxxxxxxx; xen-devel@xxxxxxxxxxxxxxxxxxxx; Rangasamy, Devaraj 
<Devaraj.Rangasamy@xxxxxxx>; Pandeshwara krishna, Mythri 
<Mythri.Pandeshwarakrishna@xxxxxxx>; SK, SivaSangeetha (Siva Sangeetha) 
<SivaSangeetha.SK@xxxxxxx>; Thomas, Rijo-john <Rijo-john.Thomas@xxxxxxx>; 
jgross@xxxxxxxx
Subject: RE: Linux pin_user_pages_fast fails on Xen

The problem is that drivers/xen/privcmd.c:privcmd_mmap sets VM_IO | VM_PFNMAP, 
and either flag would cause check_vma_flags to return -EFAULT.

Do you know if it works if you remove VM_IO | VM_PFNMAP from privcmd_mmap?

Juergen, do you think the flags are necessary and useful? Any suggestions?


On Mon, 12 Sep 2022, NK, JESHWANTHKUMAR (JESHWANTH KUMAR) wrote:
> Missed to update the Flag details:
> 
> Flag for DMA Mapped VA - 0x0C0644BB
> Flag for Local VA        -  0x08100073
> 
> 
> VM_IO and VM_PFNMAP  - Set in DMA mapped VA but not in local VA.
> 
> Regards,
> Jeshwanth
> 
> -----Original Message-----
> From: NK, JESHWANTHKUMAR (JESHWANTH KUMAR)
> Sent: Tuesday, September 13, 2022 11:05 AM
> To: 'Stefano Stabellini' <stefano.stabellini@xxxxxxx>
> Cc: Stabellini, Stefano <stefano.stabellini@xxxxxxx>; 
> boris.ostrovsky@xxxxxxxxxx; xen-devel@xxxxxxxxxxxxxxxxxxxx; Rangasamy, 
> Devaraj <Devaraj.Rangasamy@xxxxxxx>; Pandeshwara krishna, Mythri 
> <Mythri.Pandeshwarakrishna@xxxxxxx>; SK, SivaSangeetha (Siva 
> Sangeetha) <SivaSangeetha.SK@xxxxxxx>; Thomas, Rijo-john 
> <Rijo-john.Thomas@xxxxxxx>; jgross@xxxxxxxx
> Subject: RE: Linux pin_user_pages_fast fails on Xen
> 
> [AMD Official Use Only - General]
> 
> Hi Stefano,
> 
> https://elixir.bootlin.com/linux/v5.16/source/mm/gup.c#L975 is the -EFAULT 
> returning for our current use case.
> 
> access_ok is fine.
> 
> Regards,
> Jeshwanth
> 
> -----Original Message-----
> From: Stefano Stabellini <stefano.stabellini@xxxxxxx>
> Sent: Tuesday, September 13, 2022 6:56 AM
> To: NK, JESHWANTHKUMAR (JESHWANTH KUMAR) <JESHWANTHKUMAR.NK@xxxxxxx>
> Cc: Stabellini, Stefano <stefano.stabellini@xxxxxxx>; 
> boris.ostrovsky@xxxxxxxxxx; xen-devel@xxxxxxxxxxxxxxxxxxxx; NK, 
> JESHWANTHKUMAR (JESHWANTH KUMAR) <JESHWANTHKUMAR.NK@xxxxxxx>; 
> Rangasamy, Devaraj <Devaraj.Rangasamy@xxxxxxx>; Pandeshwara krishna, 
> Mythri <Mythri.Pandeshwarakrishna@xxxxxxx>; SK, SivaSangeetha (Siva 
> Sangeetha) <SivaSangeetha.SK@xxxxxxx>; Thomas, Rijo-john 
> <Rijo-john.Thomas@xxxxxxx>; jgross@xxxxxxxx
> Subject: Re: Linux pin_user_pages_fast fails on Xen
> 
> On Sat, 10 Sep 2022, Juergen Gross wrote:
> > On 09.09.22 22:25, Stefano Stabellini wrote:
> > > On Fri, 9 Sep 2022, Juergen Gross wrote:
> > > > On 09.09.22 04:11, Stefano Stabellini wrote:
> > > > > Adding more people in CC
> > > > > 
> > > > > On Thu, 8 Sep 2022, Stefano Stabellini wrote:
> > > > > > Hi Juergen,
> > > > > > 
> > > > > > A colleague is seeing a failure on x86 in Linux Dom0. The 
> > > > > > failure is pin_user_pages_fast with addresses that 
> > > > > > correspond to foreign memory
> > > > > > pages:
> > > > > > 
> > > > > > - QEMU maps a domU address using dma_memory_map
> > > > > > (xen_map_cache)
> > > > > > - QEMU calls an IOCTL to the TEE subsystem with the Virtual Address
> > > > > >     returned by dma_memory_map
> > > > > > - Linux tee_shm_register->pin_user_pages_fast Returns -14 - 
> > > > > > drivers/tee/tee_shm.c
> > > > > > 
> > > > > > Once upon a time it used to be the case that 
> > > > > > get_user_pages_fast would fail on Xen because we didn't have 
> > > > > > a struct page corresponding to foreign memory mappings. But that 
> > > > > > hasn't been the case for years now.
> > > > > > 
> > > > > > Any other ideas why it would fail?
> > > > 
> > > > I think we can expect that access_ok() isn't failing.
> > > > 
> > > > I assume the mapping was done allowing writes (sorry for paranoia mode)?
> > >   I was told it was verified: QEMU could read and write to the VA 
> > > returned by dma_memory_map. From /proc/<qemu-pid>/maps, the VA 
> > > assigned after the mapping is pointing to /dev/xen/privcmd.
> > > 
> > > 
> > > > Other than that I'm not having enough memory management skills. 
> > > > It might be related to mmap()-ed foreign pages having 
> > > > _PAGE_SPECIAL set, though.
> > > 
> > > Do we still set PAGE_SPECIAL for foreign mapped pages? It looks 
> > > like it is not there anymore? If PAGE_SPECIAL is not there, then 
> > > they really should look like regular pages?
> > 
> > See the call of pte_mkspecial() in remap_area_pfn_pte_fn() (mmu_pv.c).
> 
> The kernel version is 5.16 and the return code is -EFAULT. Is it the 
> following -EFAULT the one that triggers?
> 
> mm/gup.c:internal_get_user_pages_fast:
> 
>       if (unlikely(!access_ok((void __user *)start, len)))
>               return -EFAULT;
>