RE: xenforeignmemory fails to map PCI device memory with "Invalid Argument" error

[Paul Durrant <xadimgnik@xxxxxxxxx>]

> -----Original Message-----
> From: Xen-devel <xen-devel-bounces@xxxxxxxxxxxxxxxxxxxx> On Behalf Of ???
> Sent: 07 June 2020 23:19
> To: xen-devel@xxxxxxxxxxxxxxxxxxxx
> Subject: xenforeignmemory fails to map PCI device memory with "Invalid 
> Argument" error
> 
> Hi xen-devel,
> 
> I'm trying to read HVM guest PCI memory from Dom0 using the xenforeignmemory
> library, but it's consistently failing with "Invalid Argument", no matter
> which PCI device I try.
> However, if I try to map in regular (non-device) memory, it seems to work 
> fine.
> 
> Do you know why I can't seem to map in guest PCI memory? Am I meant to be
> mapping in the memory in some other way instead?

This behaviour is not surprising...

> 
> (Background: I'm trying to port Looking Glass to Xen:
> https://forum.level1techs.com/t/looking-glass-on-xen/158089)
> 
> Some example code to demonstrate the problem:
> 
>     #include <stdint.h>
>     #include <stdio.h>
>     #include <sys/mman.h>
>     #include <unistd.h>
>     #include <xenforeignmemory.h>
> 
>     void map(xenforeignmemory_handle* xfm, uint32_t dom, uintptr_t addr) {
>         xen_pfn_t frame = addr / sysconf(_SC_PAGESIZE);
>         void* out = xenforeignmemory_map(xfm, dom, PROT_READ |
> PROT_WRITE, 1, &frame, NULL);
>         if (!out) {
>             printf("Failed to map Dom%u's 0x%08lx: %m\n", dom, addr);
>             return;
>         }
> 
>         printf("Dom%u 0x%08lx: 0x%08lx 0x%08lx 0x%08lx 0x%08lx\n",
>             dom, addr,
>             ((unsigned long*)out)[0],
>             ((unsigned long*)out)[1],
>             ((unsigned long*)out)[2],
>             ((unsigned long*)out)[3]
>         );
>         xenforeignmemory_unmap(xfm, out, 1);
>     }
> 
>     int main(void) {
>         xenforeignmemory_handle* xfm = xenforeignmemory_open(NULL, 0);
>         if (!xfm) {
>             perror("xenforeignmemory_open");
>             return 1;
>         }
> 
>         // Regular memory - works fine
>         map(xfm, 3, 0x10000000);
> 
>         // PCI device memory - errors with "Invalid Argument"
>         map(xfm, 3, 0xf2311000);
> 
>         xenforeignmemory_close(xfm);
>         return 0;
>     }
> 
> In this case, Dom 3's 0xf2311000 belongs to the emulated SATA device:

...since, as you say here, the address belongs to an *emulated* device. To 
emulate the device, BAR accesses need to be trapped and this is done by leaving 
'holes' in the guest P2M at those addresses such than accesses cause page 
faults, at which point the faulting instruction can be examined to determine 
the nature of the access.

  Paul

> 
>     $ sudo xl qemu-monitor-command 3 'info pci'
>     ...snip...
>       Bus  0, device   5, function 0:
>         SATA controller: PCI device 8086:2922
>           PCI subsystem 1af4:1100
>           IRQ 0.
>           BAR4: I/O at 0xc260 [0xc27f].
>           BAR5: 32 bit memory at 0xf2311000 [0xf2311fff].
>           id "ahci0"
>     ...snip...