Re: SecureBoot and PCI passthrough with kernel lockdown in place (on Xen)

[Jan Beulich <jbeulich@xxxxxxxx>]

On 14.02.2022 16:02, Dario Faggioli wrote:
> We have run into an issue when trying to use PCI passthrough for a Xen
> VM running on an host where dom0 kernel is 5.14.21 (but we think it
> could be any kernel > 5.4) and SecureBoot is enabled.
> 
> The error we get, when (for instance) trying to attach a device to an
> (HVM) VM, on such system is:
> 
> # xl pci-attach 2-fv-sles15sp4beta2 0000:58:03.0 
> libxl: error: libxl_qmp.c:1838:qmp_ev_parse_error_messages: Domain 12:Failed 
> to initialize 12/15, type = 0x1, rc: -1
> libxl: error: libxl_pci.c:1777:device_pci_add_done: Domain 
> 12:libxl__device_pci_add failed for PCI device 0:58:3.0 (rc -28)
> libxl: error: libxl_device.c:1420:device_addrm_aocomplete: unable to add 
> device
> 
> QEMU, is telling us the following:
> 
> [00:04.0] xen_pt_msix_init: Error: Can't open /dev/mem: Operation not 
> permitted
> [00:04.0] xen_pt_msix_size_init: Error: Internal error: Invalid 
> xen_pt_msix_init.
> 
> And the kernel reports this:
> 
> Jan 27 16:20:53 narvi-sr860v2-bps-sles15sp4b2 kernel: Lockdown: 
> qemu-system-i38: /dev/mem,kmem,port is restricted; see man kernel_lockdown.7
> 
> So, it's related to lockdown. Which AFAIUI it's consistent with the
> fact that the problem only shows up when SecureBoot is enabled, as
> that's implies lockdown. It's also consistent with the fact that we
> don't seem to have any problems doing the same with a 5.3.x dom0
> kernel... As there's no lockdown there!
> 
> Some digging revealed that QEMU tries to open /dev/mem in
> xen_pt_msix_init():
> 
>     fd = open("/dev/mem", O_RDWR);
>     ...
>     msix->phys_iomem_base =
>             mmap(NULL,
>                  total_entries * PCI_MSIX_ENTRY_SIZE + 
> msix->table_offset_adjust,
>                  PROT_READ,
>                  MAP_SHARED | MAP_LOCKED,
>                  fd,
>                  msix->table_base + table_off - msix->table_offset_adjust);
>     close(fd);

I think this is finally a clear indication that it has always been
wrong for qemu to access hardware directly like this. I see no way
around replacing this by something which isn't a bodge / layering
violation.

Jan

> This comes from commit:
> 
> commit 3854ca577dad92c4fe97b4a6ebce360e25407af7
> Author: Jiang Yunhong <yunhong.jiang@xxxxxxxxx>
> Date:   Thu Jun 21 15:42:35 2012 +0000
> 
>     Introduce Xen PCI Passthrough, MSI
>     
>     A more complete history can be found here:
>     git://xenbits.xensource.com/qemu-xen-unstable.git
>     
>     Signed-off-by: Jiang Yunhong <yunhong.jiang@xxxxxxxxx>
>     Signed-off-by: Shan Haitao <haitao.shan@xxxxxxxxx>
>     Signed-off-by: Anthony PERARD <anthony.perard@xxxxxxxxxx>
>     Acked-by: Stefano Stabellini <stefano.stabellini@xxxxxxxxxxxxx>
> 
> Now, the questions:
> - is this (i.e., PCI-Passthrough with a locked-down dom0 kernel) 
>   working for anyone? I've Cc-ed Marek, because I think I've read that 
>   QubesOS that it does on QubesOS, but I'm not sure if the situation 
>   is the same...
> - if it's working, how?
> 
> Thanks and Regards