[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH 04/37] xen: introduce an arch helper for default dma zone status


  • To: Wei Chen <Wei.Chen@xxxxxxx>
  • From: Jan Beulich <jbeulich@xxxxxxxx>
  • Date: Wed, 19 Jan 2022 08:50:12 +0100
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=suse.com; dmarc=pass action=none header.from=suse.com; dkim=pass header.d=suse.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cTnud6A1IA2hwuRO4p/GXZP+Kz5qqVc+gGNkYw3vgWo=; b=XX5cNzviWxo4Cp4TYPppFzHQPTvJ7j4NKpKhMfg8E2XRMg5YXEZYqr9wbpx6uDOkVTkZmJ743TFjpxl6RMomDlmVpCshNkGdMB6FbSEY8j0UN9az58fQdj+CU6jox2gPQqhxCNGU/NUmTMU9x15529FB0HouLRONsWfFXCAgOZQTsohv4MA8t8jGm80Uhp+KfIANqPFjgOeKP4LPmCMGI2xASz+RHQF4G8/JClm2DO070d8E3RTKdAXwycAqbhG7WiX16Gs7qqgNF8eXUyelr2oznKYszfNkbhoCphZU6rpqwBlPljcB3qNW8rOcLhKwJe8JNyuxa0CEYrbA+ijeeQ==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XyfbYjWZuhDOoYFtbm/vLtvnjAdjQGwOL/WK8SgwAlVureRK5dmjtU/l+N/vpHMUpopbXSls6uyp38KWPIH6iElpZMeCLkNDuPRCxjVqX308ZoHCjOh0Q/jZX68RGdjosXQi+nKk2WOfNN8RFw8PO30rlClf8joDBL/oOgMFymk390uDdV8nAtp4ICfBWDjJVISLl2+TgFUni+09J2MnrQkfOsQ3n5n879hrSj1dUY4pneMm8I9kybUdlqo+DTwRXz3B0TBVkvMck2L0ewXP7hGlzEWQFhqJD5aBO4gORJp9r6LpNcRUnVo7jFoe0xi4oWq6xaJ3xpltkA+M2aQxfQ==
  • Authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=suse.com;
  • Cc: Bertrand Marquis <Bertrand.Marquis@xxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxxx>, "sstabellini@xxxxxxxxxx" <sstabellini@xxxxxxxxxx>, "julien@xxxxxxx" <julien@xxxxxxx>
  • Delivery-date: Wed, 19 Jan 2022 07:50:24 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On 19.01.2022 03:49, Wei Chen wrote:
> Hi Jan,
> 
>> -----Original Message-----
>> From: Jan Beulich <jbeulich@xxxxxxxx>
>> Sent: 2022年1月18日 22:16
>> To: Wei Chen <Wei.Chen@xxxxxxx>
>> Cc: Bertrand Marquis <Bertrand.Marquis@xxxxxxx>; xen-
>> devel@xxxxxxxxxxxxxxxxxxxx; sstabellini@xxxxxxxxxx; julien@xxxxxxx
>> Subject: Re: [PATCH 04/37] xen: introduce an arch helper for default dma
>> zone status
>>
>> On 18.01.2022 10:20, Wei Chen wrote:
>>>> From: Jan Beulich <jbeulich@xxxxxxxx>
>>>> Sent: 2022年1月18日 16:16
>>>>
>>>> On 18.01.2022 08:51, Wei Chen wrote:
>>>>>> From: Jan Beulich <jbeulich@xxxxxxxx>
>>>>>> Sent: 2022年1月18日 0:11
>>>>>> On 23.09.2021 14:02, Wei Chen wrote:
>>>>>>> In current code, when Xen is running in a multiple nodes NUMA
>>>>>>> system, it will set dma_bitsize in end_boot_allocator to reserve
>>>>>>> some low address memory for DMA.
>>>>>>>
>>>>>>> There are some x86 implications in current implementation. Becuase
>>>>>>> on x86, memory starts from 0. On a multiple nodes NUMA system, if
>>>>>>> a single node contains the majority or all of the DMA memory. x86
>>>>>>> prefer to give out memory from non-local allocations rather than
>>>>>>> exhausting the DMA memory ranges. Hence x86 use dma_bitsize to set
>>>>>>> aside some largely arbitrary amount memory for DMA memory ranges.
>>>>>>> The allocations from these memory ranges would happen only after
>>>>>>> exhausting all other nodes' memory.
>>>>>>>
>>>>>>> But the implications are not shared across all architectures. For
>>>>>>> example, Arm doesn't have these implications. So in this patch, we
>>>>>>> introduce an arch_have_default_dmazone helper for arch to determine
>>>>>>> that it need to set dma_bitsize for reserve DMA allocations or not.
>>>>>>
>>>>>> How would Arm guarantee availability of memory below a certain
>>>>>> boundary for limited-capability devices? Or is there no need
>>>>>> because there's an assumption that I/O for such devices would
>>>>>> always pass through an IOMMU, lifting address size restrictions?
>>>>>> (I guess in a !PV build on x86 we could also get rid of such a
>>>>>> reservation.)
>>>>>
>>>>> On Arm, we still can have some devices with limited DMA capability.
>>>>> And we also don't force all such devices to use IOMMU. This devices
>>>>> will affect the dma_bitsize. Like RPi platform, it sets its
>> dma_bitsize
>>>>> to 30. But in multiple NUMA nodes system, Arm doesn't have a default
>>>>> DMA zone. Multiple nodes is not a constraint on dma_bitsize. And some
>>>>> previous discussions are placed here [1].
>>>>
>>>> I'm afraid that doesn't give me more clues. For example, in the mail
>>>> being replied to there I find "That means, only first 4GB memory can
>>>> be used for DMA." Yet that's not an implication from setting
>>>> dma_bitsize. DMA is fine to occur to any address. The special address
>>>> range is being held back in case in particular Dom0 is in need of such
>>>> a range to perform I/O to _some_ devices.
>>>
>>> I am sorry that my last reply hasn't given you more clues. On Arm, only
>>> Dom0 can have DMA without IOMMU. So when we allocate memory for Dom0,
>>> we're trying to allocate memory under 4GB or in the range of dma_bitsize
>>> indicated. I think these operations meet your above Dom0 special address
>>> range description. As we have already allocated memory for DMA, so I
>>> think we don't need a DMA zone in page allocation. I am not sure is that
>>> answers your earlier question?
>>
>> I view all of this as flawed, or as a workaround at best. Xen shouldn't
>> make assumptions on what Dom0 may need. Instead Dom0 should make
>> arrangements such that it can do I/O to/from all devices of interest.
>> This may involve arranging for address restricted buffers. And for this
>> to be possible, Xen would need to have available some suitable memory.
>> I understand this is complicated by the fact that despite being HVM-like,
>> due to the lack of an IOMMU in front of certain devices address
>> restrictions on Dom0 address space alone (i.e. without any Xen
>> involvement) won't help ...
>>
> 
> I agree with you that the current implementation is probably the best
> kind of workaround. Do you have some suggestions for this patch to
> address above comments? Or should I just need to modify the commit log
> to contain some of our above discussions?

Extending the description is my primary request, or else we may end up
having the same discussion every time you submit a new version. As to
improving the situation such that preferably per-arch customization
wouldn't be necessary - that's perhaps better to be thought about by
Arm folks. Otoh, as said, an x86 build with CONFIG_PV=n could probably
leverage the new hook to actually not trigger reservation.

Jan




 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.