[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH] x86/hvmloader: don't set xenpci MMIO BAR as UC in MTRR


  • To: Roger Pau Monné <roger.pau@xxxxxxxxxx>
  • From: Jan Beulich <jbeulich@xxxxxxxx>
  • Date: Mon, 2 Jun 2025 17:22:47 +0200
  • Autocrypt: addr=jbeulich@xxxxxxxx; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL
  • Cc: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>, Anthony PERARD <anthony.perard@xxxxxxxxxx>, Anthoine Bourgeois <anthoine.bourgeois@xxxxxxxxxx>, xen-devel@xxxxxxxxxxxxxxxxxxxx
  • Delivery-date: Mon, 02 Jun 2025 15:23:06 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On 02.06.2025 16:27, Roger Pau Monné wrote:
> On Mon, Jun 02, 2025 at 11:46:52AM +0200, Jan Beulich wrote:
>> On 30.05.2025 11:23, Roger Pau Monne wrote:
>>> The Xen PCI device (vendor ID 0x5853) exposed to x86 HVM guests doesn't
>>> have the functionality of a traditional PCI device.  The exposed MIO BAR is
>>> used by some guests (including Linux) as a safe place to map foreign
>>> memory, including the grant table itself.
>>>
>>> Traditionally BARs from devices have the uncacheable (UC) cache attribute
>>> from the MTRR, to ensure correct functionality of such devices.  hvmloader
>>> mimics this behaviour and sets the MTRR attributes of both the low and high
>>> PCI MMIO windows (where BARs of PCI devices reside) as UC in MTRR.
>>>
>>> This however causes performance issues for the users of the Xen PCI device
>>> BAR, as for the purposes of mapping remote memory there's no need to use
>>> the UC attribute.  On Intel systems this is worked around by using iPAT,
>>> that allows the hypervisor to force the effective cache attribute of a p2m
>>> entry regardless of the guest PAT value.  AMD however doesn't have an
>>> equivalent of iPAT, and guest PAT values are always considered.
>>>
>>> Linux commit:
>>>
>>> 41925b105e34 xen: replace xen_remap() with memremap()
>>>
>>> Attempted to mitigate this by forcing mappings of the grant-table to use
>>> the write-back (WB) cache attribute.  However Linux memremap() takes MTRRs
>>> into account to calculate which PAT type to use, and seeing the MTRR cache
>>> attribute for the region being UC the PAT also ends up as UC, regardless of
>>> the caller having requested WB.
>>>
>>> As a workaround to allow current Linux to map the grant-table as WB using
>>> memremap() special case the Xen PCI device BAR in hvmloader and don't set
>>> its cache attribute as UC.
>>
>> Can we (fully compatibly) make such a change? IOW do we know all possible
>> guests would be at least unaffected (ideally affected positively)? Imo ...
> 
> Is there any other possible usage for the xenpci MMIO BAR?

How do you / we know?

>  My
> understanding is it was introduced for this specific purpose; to
> signal a safe place to map the grant-table or foreign mappings, which
> in both cases want to have an effective WB cache attribute.

It's a hack. It's relatively easy to imagine that someone might have built
a 2nd hack on top of this 1st one.

>>>  Such workaround in hvmloader should also be
>>> paired with a fix for Linux so it attempts to change the MTRR of the Xen
>>> PCI device BAR to WB by itself.
>>>
>>> Overall, the long term solution would be to provide the guest with a safe
>>> range in the guest physical address space where mappings to foreign pages
>>> can be created.
>>
>> ... this is the only viable path.
> 
> I agree, however this will take more time to materialize IMO.  Needs a
> patch to Linux, plus possible backports, and then distros picking the
> updates.
> 
> While I agree this needs fixing in Linux, I don't see any downsides of
> doing the workaround in hvmloader also, as a faster way to get it
> deployed with just a Xen update.

The (maybe merely theoretical) downside is that we may regress something
somewhere. Hence why I think that for starters this new behavior should
be optional, default off. Later, once proven to work in practice for a
fair while, we could then consider changing the default. (Then again I
certainly realize that adding yet another control is quite a bit of extra
effort, too.)

Jan



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.