Re: [Xen-devel] question about memory allocation for driver domain

[Oleksandr Tyshchenko <oleksandr.tyshchenko@xxxxxxxxxxxxxxx>]

Hi Julien

On Thu, Feb 5, 2015 at 6:36 PM, Oleksandr Tyshchenko
<oleksandr.tyshchenko@xxxxxxxxxxxxxxx> wrote:
> On Thu, Feb 5, 2015 at 5:24 PM, Julien Grall <julien.grall@xxxxxxxxxx> wrote:
>> Hi Oleksandr,
> Hi Julien
>>
>>
>> On 05/02/2015 21:49, Oleksandr Tyshchenko wrote:
>>>
>>> On Thu, Feb 5, 2015 at 3:12 PM, Ian Campbell <Ian.Campbell@xxxxxxxxxx>
>>> wrote:
>>>>
>>>> On Wed, 2015-02-04 at 18:47 +0200, Oleksandr Tyshchenko wrote:
>>>>>
>>>>> Hi, all.
>>>>>
>>>>> We have begun to use the driver domain on OMAP5 platform.
>>>>> To make driver domain running on OMAP5 platform we need to have it
>>>>> memory 1 to 1 mapped because of lacking SMMU support on this platform.
>>>>> To satisfy this requirement we have done temporally solution which
>>>>> works but not entirely completed and looks not good from Xen
>>>>> perspective. The main question in memory allocator in Xen.
>>>>>
>>>>> We did next steps:
>>>>> 1. toolstack:
>>>>> - allow to allocate 128/256/512 Mb memory chunks
>>>>> - add ability to set rambase_pfn via cfg file
>>>>>
>>>>> 2. hypervisor:
>>>>> - alloc driver domain memory 1 to 1
>>>>>    - mark domain with id 1 as privileged
>>>>>    - update memory allocation logic for such domain
>>>>>    - allocate memory in the same way as for domain 0
>>>>>
>>>>> But we have encountered with one thing related to memory allocation
>>>>> algorithm.
>>>>> Let me describe what I mean.
>>>>> Our requirements is to allocate one specified chunk if it is present in
>>>>> domheap.
>>>>> We have this "spec_mfn" which are a "rambase_pfn" for driver domain we
>>>>> want to map 1 to 1. We can get it from extent_list. So, we need to
>>>>> alloc chunk which would be correspond to "spec_mfn".
>>>>> In other world we need to allocate known chunk of memory. But if I
>>>>> understood correctly that the existing allocator doesn't allow us to
>>>>> do that directly.
>>>>>
>>>>> There are some thoughts how to do that. But, at first, we need to
>>>>> choose right direction:
>>>>> 1. Add the separate memory allocator which allow us to alloc specified
>>>>> chunk if it is present. And use it in case when we need to allocate
>>>>> memory for driver domain 1:1 only. We can pass bool variable via cfg
>>>>> file to show what we want (1:1 or not).
>>>>> 2. Don't add separate allocator. Modify existing allocator to add
>>>>> ability to alloc specified chunk.
>>>>> 3. Don't add and modify anything. Let the allocator to work as usual.
>>>>> Return mfn of allocating chunk and correct the default rambase_pfn in
>>>>> toolstack.
>>>>> What we actually have in the moment but do that manually. We see what
>>>>> the mfn we got and corrected "rambase_pfn" property in cfg file.
>>>>>
>>>>> Could someone explain me what is a right way?
>>>>
>>>>
>>>> The approach we've taken with dom0 is not to require specific addresses,
>>>> but rather to tailor the address map to the contiguous region the
>>>> allocator gives us.
>>>>
>>>> Since I presume you are using a 1:1 layout in the driver domain too I
>>>> expect that approach should work there too (I think this is your #3?).
>>>
>>> yes
>>>>
>>>>
>>>> Your #2 might be possible, but would probably involve a reasonably
>>>> expensive scan of the various free pools in the hopes of finding the
>>>> block you want, since it isn't designed to be looked up in this way.
>>>>
>>>> I suppose #1 would be something like Linux's CMA allocator -- i.e.
>>>> carving out 1 or more regions on boot and keeping them away from the
>>>> main allocator (but still in e.g. the frametable etc) and a simple way
>>>> to allocate one of the chunks.
>>>
>>> This is interesting. Can I use /memreserve/ logic or something similar to
>>> keeping them away from the main allocator?
>>> But at the stages where we "initialize" domheap we know nothing about
>>> guest domain (domd) and how we need to allocate memory for it (1:1 or
>>> not).
>>
>>
>> The first approach we had for DOM0 1:1 mapping was book-keeping memory at
>> startup. The main allocator wasn't able to use it.
>>
>> But this turn out to be very hackish and not scalable (multiple memory
>> banks...).
>>
>>>>
>>>> So I think any of the approaches you suggest could work, I'd probably
>>>> say #3, #1, #2 in decreasing order of preference.
>>>
>>> I got it.
>>
>>
>> If I correctly understand the solution #3, it may end up to lots of small
>> banks after domains has been created/destroyed multiple
> I do not quite understand why this can happen. Anyway, I need to check
> this assumption.
> I will let you know about result.

Let me describe in detail about solution #3 before answer to your
question. Maybe I missed something in
the first mail. Also the Ian's answer clarified to me some points.
We don't have complete solution for now. We only have temporally
solution in which
we relied on assumptions which might or not be acceptable, but it
seems that the approach in general could work.
If it is true the our target is to rewrite/rework this stuff, to make
it more cleaner and correct from XEN point of view for platforms which
doesn't have a SMMU
even in the case where this stuff never reaches upstream.

To run driver domain (domd) on OMAP5 platform with 1:1 mapping we did
next preparations:
(here I try to explain about memory allocation only, IRQ handling and
other things are out of scope for the current thread)
1. Since the domd can use 128/256/512 Mb of RAM we modified existing
populate_guest_memory() in xc_dom_arm.c to allow to allocate
128/256/512 Mb memory chunks.
2. Since the default rambase (0x40000000) doesn't suitable for us for
some reasons:
- OMAP5 platform has memory mapped registers, starting from 0x4800000
- We have 2 guest domains (domd and domU) so it should be different
rambases for them
we added ability for toolstack to pass it via domain config file.
3. Since for domd we need one contiguous chunk of memory we created a
new function allocate_domd_memory_11() in common/memory.c for the next
purposes:
- To allocate one contiguous memory chunk with specified order (as it
was done for dom0)
- To add this allocated chunk to the guest - domd (via
guest_physmap_add_page(), taking into account that mfn=gpfn)
4. Since we need to allocate memory before any operation with it we
created hook for XENMEM_populate_physmap command. Here we relied on
the assumption
that the domain_id for domd is always 1 (it is not true after the domd
has been created again after destroying).
During first XENMEM_populate_physmap command we set is_privileged=true
and call allocate_domd_memory_11(),
during next commands - call populate_physmap() as a usual. The
is_domain_direct_mapped condition is a our case in populate_physmap().
I know that it is a very, very hacking solution). But, let's continue...

How it works at the moment?
1. Create domd with default rambase_pfn (0x80000000).
2. See what the mfn we got in allocate_domd_memory_11().
3. Set rambase_pfn=mfn in config file.
If the system configuration (N domains, domains memory, etc.) is not
changed, we will always get the same mfn. If we decided to change
something, for example, domd memory we need to repeat steps 2 and 3.
Yes, looks not good.

How it should works?
The approach is to tailor the domd address map to the contiguous
region the allocator gives us. So, a guest rambase (gpfn) must based
on
mfn of a page we have allocated successfully. Without any manual actions.

I think it must be done in next steps:
1. Add a separate command, XENMEM_alloc_memory_11 or something similar
and it should be issued before call xc_dom_rambase_init() in
libxl_dom.c in case of presence
domd_memory_11 property in config file only. This should remove
terrible hook and anything related to d->domain_id=1 in
common/memory.c.
2. Pass returned by XENMEM_alloc_memory_11 result to the xc_dom_rambase_init().

What are the advantages in compare with solution #1 and solution #2.
1. There is no need to add standalone allocator or modifying existing.

Let's return to your question about created/destroyed domd multiple times.
I have tried to do this with solution we have at the moment. I added
some modifications to allow me to destroy/create domd multiple times.
And I have seen that the allocator always returns the same page. This
means that the all memory allocated for domd have been returned to the
heap. Am I right?
Or perhaps did you mean that this may happens with the completed solution?

>
>>
>>>>
>>>> Now, if you were asking for ideas on how to make this stuff
>>>> upstreamable, well that's a harder question. I'm not really sure :-/
>>>
>>> Ideally, I would like to make this stuff upstreamable.
>>
>>
>> As you said on your first mail, this is required to allow DMA on device
>> passthrough when the platform doesn't have an SMMU.
>>
>> I totally understand a such use case for specific embedded product, because
>> you trust your driver domains.
>>
>> On upstream, the main use case of driver domain is too protected the
>> platform from buggy drivers. If the drivers crash, then you can restart only
>> this domain.
>>
>> With a device protected by an SMMU, any DMA requests are safe. Without it,
>> the driver can pretty much do whatever it wants. This could lead to the
>> hypervisor corruption/crash.
>>
>> If a such feature as to come to Xen upstream, this should be hidden to the
>> main users (maybe via compilation option) or require a specific option to
>> use it.
> Completely agree
>
>>
>> Regards.
>>
>> --
>> Julien Grall
>
>
>
> --
>
> Oleksandr Tyshchenko | Embedded Dev
> GlobalLogic
> www.globallogic.com



-- 

Oleksandr Tyshchenko | Embedded Dev
GlobalLogic
www.globallogic.com

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel