Re: [RFC PATCH] xen/memory: Introduce a hypercall to provide unallocated space

[Oleksandr <olekstysh@xxxxxxxxx>]

On 09.08.21 18:42, Julien Grall wrote:
> Hi Oleksandr,

Hi Julien.


Thank you for the input.


> On 07/08/2021 18:03, Oleksandr wrote:
> > On 06.08.21 03:30, Stefano Stabellini wrote:
> >
> > Hi Stefano
> >
> > > On Wed, 4 Aug 2021, Julien Grall wrote:
> > > > > +#define GUEST_SAFE_RANGE_BASE xen_mk_ullong(0xDE00000000) /* 
> > > > > 128GB */
> > > > > +#define GUEST_SAFE_RANGE_SIZE xen_mk_ullong(0x0200000000)
> > > > >
> > > > > While the possible new DT bindings has not been agreed yet, I re-used
> > > > > existing "reg" property under the hypervisor node to pass safe 
> > > > > range as a
> > > > > second region,
> > > > > https://elixir.bootlin.com/linux/v5.14-rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10: 
> > > > So a single region works for a guest today, but for dom0 we will 
> > > > need multiple
> > > > regions because it is may be difficult to find enough contiguous 
> > > > space for a
> > > > single region.
> > > >
> > > > That said, as dom0 is mapped 1:1 (including some guest mapping), 
> > > > there is also
> > > > the question where to allocate the safe region. For grant table, we 
> > > > so far
> > > > re-use the Xen address space because it is assumed it will space 
> > > > will always
> > > > be bigger than the grant table.
> > > >
> > > > I am not sure yet where we could allocate the safe regions. 
> > > > Stefano, do you
> > > > have any ideas?
> > > The safest choice would be the address range corresponding to memory
> > > (/memory) not already allocated to Dom0.
> > >
> > > For instance from my last boot logs:
> > > (XEN) Allocating 1:1 mappings totalling 1600MB for dom0:
> > > (XEN) BANK[0] 0x00000010000000-0x00000070000000 (1536MB)
> > > (XEN) BANK[1] 0x00000078000000-0x0000007c000000 (64MB)
> > >
> > > All the other ranges could be given as unallocated space:
> > >
> > > - 0x0 - 0x10000000
> > > - 0x70000000 - 0x78000000
> > > - 0x8_0000_0000 - 0x8_8000_0000
> > Thank you for the ideas.
> >
> > If I got the idea correctly, yes, as these ranges represent the real 
> > RAM, so no I/O would be in conflict with them and as the result - no 
> > overlaps would be expected.
> > But, I wonder, would this work if we have IOMMU enabled for Dom0 and 
> > need to establish 1:1 mapping for the DMA devices to work with grant 
> > mappings...
> > In arm_iommu_map_page() we call guest_physmap_add_entry() with gfn = 
> > mfn, so the question is could we end up with this new gfn replacing 
> > the valid mapping
> > (with gfn allocated from the safe region)?
> Right, when we enable the IOMMU for dom0, Xen will add an extra 
> mapping with GFN == MFN for foreign and grant pages. This is because 
> Linux is not aware that whether a device is protected by an IOMMU. 
> Therefore it is assuming it is not and will use the MFN to configure 
> for DMA transaction.
> We can't remove the mapping without significant changes in Linux and 
> Xen. I would not mandate them for this work.
> That said, I think it would be acceptable to have different way to 
> find the region depending on the dom0 configuration. So we could use 
> the RAM not used by dom0 when the IOMMU is turned off.
OK


> > > The second best choice would be an hole: an address range not used by
> > > anybody else (no reg property) and also not even mappable by a bus (not
> > > covered by a ranges property). This is not the best choice because 
> > > there
> > > can cases where physical resources appear afterwards.
> Are you saying that the original device-tree doesn't even describe 
> them in any way (i.e. reserved...)?
> > Unfortunately, yes.
> So the decision where the safe region is located will be done by Xen. 
> There is no involvement of the domain (it will discover the region 
> from the DT). Therefore, I don't think we need to think about 
> everything right now as we could adapt this is exact region is not 
> part of the stable ABI.
> The hotplug is one I would defer because this is not supported (and 
> quite likely not working) in Xen upstream today.
Sounds reasonable.


> Now regarding the case where dom0 is using the IOMMU. The assumption 
> is Xen will be able to figure out all the regions used from the 
> firmware table (ACPI or DT).
> AFAIK, this assumption would be correct for DT. However, for ACPI, I 
> remember we were not able to find all the MMIOs region in Xen (see [1] 
> and [2]). So even this solution would not work for ACPI.
> If I am not mistaken, we don't support IOMMU with ACPI yet. So we 
> could defer the problem to when this is going to be supported.
Sounds reasonable.


To summarize:

0. Skip ACPI case for now, implement for DT case

1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as 
safe ranges
I would take into the account holes >= 1MB. I am wondering, do we need a 
special alignment here other than a PAGE_SIZE?
2. If IOMMU is disabled for Dom0 -> provide RAM which not assigned to 
Dom0 as safe ranges
We could even provide holes here as well.


> Cheers,
>
> [1] https://marc.info/?l=linux-arm-kernel&m=148469169210500&w=2
> [2] Xen commit 80f9c316708400cea4417e36337267d3b26591db
--
Regards,

Oleksandr Tyshchenko