xenforeignmemory fails to map PCI device memory with "Invalid Argument" error

[‍小太 <nospam@xxxxxxxx>]

Hi xen-devel,

I'm trying to read HVM guest PCI memory from Dom0 using the xenforeignmemory
library, but it's consistently failing with "Invalid Argument", no matter
which PCI device I try.
However, if I try to map in regular (non-device) memory, it seems to work fine.

Do you know why I can't seem to map in guest PCI memory? Am I meant to be
mapping in the memory in some other way instead?

(Background: I'm trying to port Looking Glass to Xen:
https://forum.level1techs.com/t/looking-glass-on-xen/158089)

Some example code to demonstrate the problem:

    #include <stdint.h>
    #include <stdio.h>
    #include <sys/mman.h>
    #include <unistd.h>
    #include <xenforeignmemory.h>

    void map(xenforeignmemory_handle* xfm, uint32_t dom, uintptr_t addr) {
        xen_pfn_t frame = addr / sysconf(_SC_PAGESIZE);
        void* out = xenforeignmemory_map(xfm, dom, PROT_READ |
PROT_WRITE, 1, &frame, NULL);
        if (!out) {
            printf("Failed to map Dom%u's 0x%08lx: %m\n", dom, addr);
            return;
        }

        printf("Dom%u 0x%08lx: 0x%08lx 0x%08lx 0x%08lx 0x%08lx\n",
            dom, addr,
            ((unsigned long*)out)[0],
            ((unsigned long*)out)[1],
            ((unsigned long*)out)[2],
            ((unsigned long*)out)[3]
        );
        xenforeignmemory_unmap(xfm, out, 1);
    }

    int main(void) {
        xenforeignmemory_handle* xfm = xenforeignmemory_open(NULL, 0);
        if (!xfm) {
            perror("xenforeignmemory_open");
            return 1;
        }

        // Regular memory - works fine
        map(xfm, 3, 0x10000000);

        // PCI device memory - errors with "Invalid Argument"
        map(xfm, 3, 0xf2311000);

        xenforeignmemory_close(xfm);
        return 0;
    }

In this case, Dom 3's 0xf2311000 belongs to the emulated SATA device:

    $ sudo xl qemu-monitor-command 3 'info pci'
    ...snip...
      Bus  0, device   5, function 0:
        SATA controller: PCI device 8086:2922
          PCI subsystem 1af4:1100
          IRQ 0.
          BAR4: I/O at 0xc260 [0xc27f].
          BAR5: 32 bit memory at 0xf2311000 [0xf2311fff].
          id "ahci0"
    ...snip...