Re: [PATCH v5] x86/hvmloader: select xen platform pci MMIO BAR UC or WB MTRR cache attribute

[Anthony PERARD <anthony@xxxxxxxxxxxxxx>]

On Fri, Jun 13, 2025 at 01:00:09PM +0200, Roger Pau Monne wrote:
> The Xen platform PCI device (vendor ID 0x5853) exposed to x86 HVM guests
> doesn't have the functionality of a traditional PCI device.  The exposed
> MMIO BAR is used by some guests (including Linux) as a safe place to map
> foreign memory, including the grant table itself.
> 
> Traditionally BARs from devices have the uncacheable (UC) cache attribute
> from the MTRR, to ensure correct functionality of such devices.  hvmloader
> mimics this behavior and sets the MTRR attributes of both the low and high
> PCI MMIO windows (where BARs of PCI devices reside) as UC in MTRR.
> 
> This however causes performance issues for users of the Xen platform PCI
> device BAR, as for the purposes of mapping remote memory there's no need to
> use the UC attribute.  On Intel systems this is worked around by using
> iPAT, that allows the hypervisor to force the effective cache attribute of
> a p2m entry regardless of the guest PAT value.  AMD however doesn't have an
> equivalent of iPAT, and guest PAT values are always considered.
> 
> Linux commit:
> 
> 41925b105e34 xen: replace xen_remap() with memremap()
> 
> Attempted to mitigate this by forcing mappings of the grant-table to use
> the write-back (WB) cache attribute.  However Linux memremap() takes MTRRs
> into account to calculate which PAT type to use, and seeing the MTRR cache
> attribute for the region being UC the PAT also ends up as UC, regardless of
> the caller having requested WB.
> 
> As a workaround to allow current Linux to map the grant-table as WB using
> memremap() introduce an xl.cfg option (xen_platform_pci_bar_uc=0) that can
> be used to select whether the Xen platform PCI device BAR will have the UC
> attribute in MTRR.  Such workaround in hvmloader should also be paired with
> a fix for Linux so it attempts to change the MTRR of the Xen platform PCI
> device BAR to WB by itself.
> 
> Overall, the long term solution would be to provide the guest with a safe
> range in the guest physical address space where mappings to foreign pages
> can be created.
> 
> Some vif throughput performance figures provided by Anthoine from a 8
> vCPUs, 4GB of RAM HVM guest(s) running on AMD hardware:
> 
> Without this patch:
> vm -> dom0: 1.1Gb/s
> vm -> vm:   5.0Gb/s
> 
> With the patch:
> vm -> dom0: 4.5Gb/s
> vm -> vm:   7.0Gb/s
> 
> Reported-by: Anthoine Bourgeois <anthoine.bourgeois@xxxxxxxxxx>
> Signed-off-by: Roger Pau Monné <roger.pau@xxxxxxxxxx>
> Reviewed-by: Oleksii Kurochko<oleksii.kurochko@xxxxxxxxx>
> Acked-by: Jan Beulich <jbeulich@xxxxxxxx> # hvmloader
> ---
> Changes since v4:
>  - Rename to Xen platform PCI to avoid confusion.
>  - Set hvmloader BAR default to UC.
>  - Unconditionally write XS node in libxl.
>  - Introduce define for Xen PCI vendor ID.

Reviewed-by: Anthony PERARD <anthony.perard@xxxxxxxxxx>

Thanks,

-- 
Anthony PERARD