[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: [PATCH 04/37] xen: introduce an arch helper for default dma zone status


  • To: Jan Beulich <jbeulich@xxxxxxxx>
  • From: Wei Chen <Wei.Chen@xxxxxxx>
  • Date: Wed, 19 Jan 2022 08:33:41 +0000
  • Accept-language: en-US
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6qwy9k64ifZJ0zdhC3c3QPVRCXjZcEOI7lHqezt0o54=; b=gENRhZk9zuY3jq/ml8RX5fzSgSCgSiG/xghPTIOTFd7JFMPhhiERweiHGHpWLu7dWPswwyOGkFex+e5J+qP6zA9NHaqhnIxCE0ySdlTjpnTbdCd2hOlr8mSXD4KJ+uxhtv1rs+7LlJ7MBD6S6tuIbKlD4TF/qs8Osn6oiBUZcwws7YUSBRNSnjhaKlN0k9RB8k5qOycf7OGCW8mD0YArXgI7qsXJPjdHu5sU1y7I4wnflf78X+vkp+GaLzdZtCMxjlJY0mBMkpJNy48Ce9uaKcnIOChCWtM8rcjnor8sVQzIUaj1/KuhG1LpgyVDFCSYmzO6EvjWBxsnlzUoHZqriA==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UtKooMe9HqNjHCKD7Rsj37JMIf4Dq9vtRG5QRrB/rnIVjXEVx05mLuxE5Tyn6hJZzk1bt6e30UiCnn6bxD/s51jxVDUEOOSNRdXwmWzksjqq1FdKkyivfKjynqbShkg93NPuGIULrL8mnFiTKvdY5Uyg//GH0a95/gvI4lHPW374b/Um/btXFvflBmXrCy6sp22BMuNRryKPeNLNnpR7gWBYu6RK4xdzVl07bXU3pcPzqMGFEdBtxayrKYeKQv1vK/0WkqqKVqtnMpEcmKemMs1Zl0Lje3i4OypTj33zoiTNVlD4odqz15Ch8eQD+I2+6WWGorWMR4ULMBxqcAV+Ww==
  • Authentication-results-original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com;
  • Cc: Bertrand Marquis <Bertrand.Marquis@xxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxxx>, "sstabellini@xxxxxxxxxx" <sstabellini@xxxxxxxxxx>, "julien@xxxxxxx" <julien@xxxxxxx>
  • Delivery-date: Wed, 19 Jan 2022 08:34:01 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Nodisclaimer: true
  • Original-authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com;
  • Thread-index: AQHXsHMGx8xGhif/FUCDSa9NS/fSLqxoGKiAgAD49HCAABTIAIAACXAAgABbKICAANEXUIAAVWcAgAAL52A=
  • Thread-topic: [PATCH 04/37] xen: introduce an arch helper for default dma zone status

Hi Jan,

> -----Original Message-----
> From: Jan Beulich <jbeulich@xxxxxxxx>
> Sent: 2022年1月19日 15:50
> To: Wei Chen <Wei.Chen@xxxxxxx>
> Cc: Bertrand Marquis <Bertrand.Marquis@xxxxxxx>; xen-
> devel@xxxxxxxxxxxxxxxxxxxx; sstabellini@xxxxxxxxxx; julien@xxxxxxx
> Subject: Re: [PATCH 04/37] xen: introduce an arch helper for default dma
> zone status
> 
> On 19.01.2022 03:49, Wei Chen wrote:
> > Hi Jan,
> >
> >> -----Original Message-----
> >> From: Jan Beulich <jbeulich@xxxxxxxx>
> >> Sent: 2022年1月18日 22:16
> >> To: Wei Chen <Wei.Chen@xxxxxxx>
> >> Cc: Bertrand Marquis <Bertrand.Marquis@xxxxxxx>; xen-
> >> devel@xxxxxxxxxxxxxxxxxxxx; sstabellini@xxxxxxxxxx; julien@xxxxxxx
> >> Subject: Re: [PATCH 04/37] xen: introduce an arch helper for default
> dma
> >> zone status
> >>
> >> On 18.01.2022 10:20, Wei Chen wrote:
> >>>> From: Jan Beulich <jbeulich@xxxxxxxx>
> >>>> Sent: 2022年1月18日 16:16
> >>>>
> >>>> On 18.01.2022 08:51, Wei Chen wrote:
> >>>>>> From: Jan Beulich <jbeulich@xxxxxxxx>
> >>>>>> Sent: 2022年1月18日 0:11
> >>>>>> On 23.09.2021 14:02, Wei Chen wrote:
> >>>>>>> In current code, when Xen is running in a multiple nodes NUMA
> >>>>>>> system, it will set dma_bitsize in end_boot_allocator to reserve
> >>>>>>> some low address memory for DMA.
> >>>>>>>
> >>>>>>> There are some x86 implications in current implementation. Becuase
> >>>>>>> on x86, memory starts from 0. On a multiple nodes NUMA system, if
> >>>>>>> a single node contains the majority or all of the DMA memory. x86
> >>>>>>> prefer to give out memory from non-local allocations rather than
> >>>>>>> exhausting the DMA memory ranges. Hence x86 use dma_bitsize to set
> >>>>>>> aside some largely arbitrary amount memory for DMA memory ranges.
> >>>>>>> The allocations from these memory ranges would happen only after
> >>>>>>> exhausting all other nodes' memory.
> >>>>>>>
> >>>>>>> But the implications are not shared across all architectures. For
> >>>>>>> example, Arm doesn't have these implications. So in this patch, we
> >>>>>>> introduce an arch_have_default_dmazone helper for arch to
> determine
> >>>>>>> that it need to set dma_bitsize for reserve DMA allocations or not.
> >>>>>>
> >>>>>> How would Arm guarantee availability of memory below a certain
> >>>>>> boundary for limited-capability devices? Or is there no need
> >>>>>> because there's an assumption that I/O for such devices would
> >>>>>> always pass through an IOMMU, lifting address size restrictions?
> >>>>>> (I guess in a !PV build on x86 we could also get rid of such a
> >>>>>> reservation.)
> >>>>>
> >>>>> On Arm, we still can have some devices with limited DMA capability.
> >>>>> And we also don't force all such devices to use IOMMU. This devices
> >>>>> will affect the dma_bitsize. Like RPi platform, it sets its
> >> dma_bitsize
> >>>>> to 30. But in multiple NUMA nodes system, Arm doesn't have a default
> >>>>> DMA zone. Multiple nodes is not a constraint on dma_bitsize. And
> some
> >>>>> previous discussions are placed here [1].
> >>>>
> >>>> I'm afraid that doesn't give me more clues. For example, in the mail
> >>>> being replied to there I find "That means, only first 4GB memory can
> >>>> be used for DMA." Yet that's not an implication from setting
> >>>> dma_bitsize. DMA is fine to occur to any address. The special address
> >>>> range is being held back in case in particular Dom0 is in need of
> such
> >>>> a range to perform I/O to _some_ devices.
> >>>
> >>> I am sorry that my last reply hasn't given you more clues. On Arm,
> only
> >>> Dom0 can have DMA without IOMMU. So when we allocate memory for Dom0,
> >>> we're trying to allocate memory under 4GB or in the range of
> dma_bitsize
> >>> indicated. I think these operations meet your above Dom0 special
> address
> >>> range description. As we have already allocated memory for DMA, so I
> >>> think we don't need a DMA zone in page allocation. I am not sure is
> that
> >>> answers your earlier question?
> >>
> >> I view all of this as flawed, or as a workaround at best. Xen shouldn't
> >> make assumptions on what Dom0 may need. Instead Dom0 should make
> >> arrangements such that it can do I/O to/from all devices of interest.
> >> This may involve arranging for address restricted buffers. And for this
> >> to be possible, Xen would need to have available some suitable memory.
> >> I understand this is complicated by the fact that despite being HVM-
> like,
> >> due to the lack of an IOMMU in front of certain devices address
> >> restrictions on Dom0 address space alone (i.e. without any Xen
> >> involvement) won't help ...
> >>
> >
> > I agree with you that the current implementation is probably the best
> > kind of workaround. Do you have some suggestions for this patch to
> > address above comments? Or should I just need to modify the commit log
> > to contain some of our above discussions?
> 
> Extending the description is my primary request, or else we may end up
> having the same discussion every time you submit a new version. As to
> improving the situation such that preferably per-arch customization
> wouldn't be necessary - that's perhaps better to be thought about by
> Arm folks. Otoh, as said, an x86 build with CONFIG_PV=n could probably
> leverage the new hook to actually not trigger reservation.
> 

Ok, I will try to extend the description to avoid the same discussion
in next version.

> Jan


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.