Re: [Xen-devel] Proposal to allow setting up shared memory areas between VMs from xl config file

[Zhongze Liu <blackskygg@xxxxxxxxx>]

Hi, Wei,

>>>>2017-05-19 17:33 GMT+08:00 Wei Liu <wei.liu2@xxxxxxxxxx>:
> On Thu, May 18, 2017 at 11:09:40AM -0700, Stefano Stabellini wrote:
>> > >
>> > > Yes, those attributes are necessary and should be explicitly specified 
>> > > in the
>> > > config file. I'll add them in the next version of this proposal. And 
>> > > taking the
>> > > granularity into consideration, what do you say if we change the entries 
>> > > into
>> > > something like:
>> > > 'start=0xcafebabe, end=0xdeedbeef, granularity=4K, prot=RWX'.
>> >
>> > I realised I may have gone too far after reading your reply.
>> >
>> > What is the end purpose of this project? If you only want to insert a
>> > mfn into guest address space and don't care how the guest is going to
>> > map it, you can omit the prot= part. If you want stricter control, you
>> > will need them -- and that would also have implications on the
>> > hypervisor code you need.
>> >
>> > I suggest you write the manual for the new mechanism you propose first.
>> > That way you describe the feature in a sysadmin-friendly way.  Describe
>> > the syntax, the effect of the new mechanism and how people are supposed
>> > to use it under what circumstances.
>>
>> The memory sharing mechanism should enable guests to communicate with
>> each other using a shared ring. That implies that the memory needs to be
>> read-write, but I can imagine there are use cases for it to be read-only
>> too. I think it is a good idea to specify it.
>>
>> However, I do not think we should ask Zhongze to write a protocol
>> specification for how these guests should communicate. That is out of
>> scope.
>
> That's right. This is out of scope. I didn't mean to ask Zhongze to
> write a protocol specification.
>


Yes, describing the syntax and the effect of the new mechanism, as well as
example usecases, will be of great importance. I'm now waiting for Stefano to
get some real llfe customer usecases.

>
>>
>>
>> > > >> In the example above. A memory area A1 will be shared between
>> > > >> vm1 and vm2 -- vm1 can access this area using addr_range1
>> > > >> and vm2 using addr_range3. Likewise, a memory area A2 will be
>> > > >> shared between vm1 and vm3 -- vm1 can access A2 using addr_range2
>> > > >> and vm3 using addr_range4.
>> > > >>
>> > > >> The shared memory area denoted by an identifier IDx will be
>> > > >> allocated when it first appear, and the memory pages will be taken 
>> > > >> from
>> > > >> the first VM whose static_shared_mem list contains IDx. Take the above
>> > > >> config files for example, if we instantiate vm1, vm2 and vm3, one 
>> > > >> after
>> > > >> another, the memory areas denoted by ID1 and ID2 will both be 
>> > > >> allocated
>> > > >> in and taken from vm1.
>> > > >
>> > > > Hmm... I can see some potential hazards. Currently, multiple xl 
>> > > > processes
>> > > > are serialized by a lock, and your assumption is the creation is done 
>> > > > in
>> > > > order, but suppose sometime later they can run in parallel. When you
>> > > > have several "xl create" and they race with each other, what will
>> > > > happen?
>> > > >
>> > > > This can be solved by serializing in libxl or hypervisor, I think.
>> > > > It is up to you to choose where to do it.
>> > > >
>> > > > Also, please consider what happens when you destroy the owner domain
>> > > > before the rest. Proper reference counting should be done in the
>> > > > hypervisor.
>> > > >
>> > >
>> > > Yes, the access to xenstore and other shared data should be serialized
>> > > using some kind of lock.
>> > >
>> > > >
>> > > >>
>> > > >> ======================================
>> > > >> 2.2 Store the mem-sharing information in xenstore
>> > > >> ======================================
>> > > >> This information should include the length and owner of the area. And
>> > > >> it should also include information about where the backing memory 
>> > > >> areas
>> > > >> are mapped in every VM that are using it. This information should be
>> > > >> known to the xl command and all domains, so we utilize xenstore to 
>> > > >> keep
>> > > >> this information. A current plan is to place the information under
>> > > >> /local/shared_mem/ID. Still take the above config files as an example:
>> > > >>
>> > > >> If we instantiate vm1, vm2 and vm3, one after another,
>> > > >> “xenstore ls -f” should output something like this:
>> > > >>
>> > > >>
>> > > >> After VM1 was instantiated, the output of “xenstore ls -f”
>> > > >> will be something like this:
>> > > >>
>> > > >>     /local/shared_mem/ID1/owner = dom_id_of_vm1
>> > > >>
>> > > >>     /local/shared_mem/ID1/size = sizeof_addr_range1
>> > > >>
>> > > >>     /local/shared_mem/ID1/mappings/dom_id_of_vm1 = addr_range1
>> > > >>
>> > > >>
>> > > >>     /local/shared_mem/ID2/owner = dom_id_of_vm1
>> > > >>
>> > > >>     /local/shared_mem/ID2/size = sizeof_addr_range1
>> > > >>
>> > > >>     /local/shared_mem/ID2/mappings/dom_id_of_vm1 = addr_range2
>> > > >>
>> > > >>
>> > > >> After VM2 was instantiated, the following new lines will appear:
>> > > >>
>> > > >>     /local/shared_mem/ID1/mappings/dom_id_of_vm2 = addr_range3
>> > > >>
>> > > >>
>> > > >> After VM2 was instantiated, the following new lines will appear:
>> > > >>
>> > > >>     /local/shared_mem/ID2/mappings/dom_id_of_vm2 = addr_range4
>> > > >>
>> > > >> When we encounter an id IDx during "xl create":
>> > > >>
>> > > >>   + If it’s not under /local/shared_mem, create the corresponding 
>> > > >> entries
>> > > >>      (owner, size, and mappings) in xenstore, and allocate the memory 
>> > > >> from
>> > > >>      the newly created domain.
>> > > >>
>> > > >>   + If it’s found under /local/shared_mem, map the pages to the newly
>> > > >>       created domain, and add the current domain to
>> > > >>       /local/shared_mem/IDx/mappings.
>> > > >>
>> > > >
>> > > > Again, please think about destruction as well.
>> > > >
>> > > > At this point I think modelling after POSIX shared memory makes more
>> > > > sense. That is, there isn't one "owner" for the memory. You get hold of
>> > > > the shared memory via a key (ID in your case?).
>> > > >
>> > >
>> > > Actually, I've thought about the same question and have discussed this 
>> > > with
>> > > Julien and Stefano. And this what they told me:
>> > >
>> > > Stefano wrote:
>> > > "I think that in your scenario Xen (the hypervisor) wouldn't allow the
>> > > first domain to be completely destroyed because it knows that its
>> > > memory is still in use by something else in the system. The domain
>> > > remains in a zombie state until the memory is not used anymore. We need
>> > > to double-check this, but I don't think it will be a problem."
>> > >
>> >
>> > This has security implications -- a rogue guest can prevent the
>> > destruction of the owner.
>>
>> We are going to use the same underlying hypervisor infrastructure, the
>> end result should be no different than sharing memory via grant table
>> from a security perspective. If not, then we need to fix Xen.
>>
>
> There is a certain level of trust in the frontend / backend model. The
> frontend needs to trust backend to a certain degree. A user knows what
> to expect or do if one side misbehaves.
>
> But the way this proposal is phrased is that this is to construct a
> communication channel, i.e. it reads to me from a user's perspective you
> don't give one guest more trust than the other. This needs clarifying.
>

IMHO, Since the shared memory is statically specified in the xl config files,
VMs don't have to ability to dynamically ask for a shared page from other
VMS through this new mechanism,
 The xl file is written by the sys admin, so I think it's the administrators'
duty to make it right.

>
>>
>> > > and Julien wrote:
>> > > "That's correct. A domain will not be destroyed until all the memory
>> > > associated to it will be freed.
>> > > A page will be considered free when all the reference on it will be
>> > > removed. This means that if the domain who allocated the page die, it
>> > > will not be fully destroyed until the page is not used by another
>> > > domain.
>> > > This is assuming that every domain using the page is taking a
>> > > reference (similar to foreign mapping). Actually, I think we might be
>> > > able to re-use the mapspace XENMAPSPACE_gmfn_foreign.
>> > > Actually, I think we can re-use the same mechanism as foreign mapping 
>> > > (see
>> > > Note that Xen on ARM (and x86?) does not take reference when mapping a
>> > > page to a stage-2 page table (e.g the page table holding the
>> > > translation between a guest physical address and host physical
>> > > address)."
>> > >
>> > > I've also thought about modeling after the POSIX way of sharing memory.
>> > > If we do so, the owner of the shared pages should be Dom0, and we
>> > > will have to do the reference counting ourselves, and free pages when 
>> > > they're
>> > > no longer needed. I'm not sure which method is better. What do you say?
>> > >
>> >
>> > Assigning the page to Dom0 doesn't sound right to me either.
>

Then do you have any ideas here?

>
>> >
>> > But the first step should really be defining the scope of the project.
>> > Technical details will follow naturally.
>>
>> I thought that Zhongze wrote it well in "Motivation and Description".
>> What would you like to know in addition to that?
>
> A bit more details are needed. See above.

I'll try to take all the good points  in the comments into consideration.
Thank you.

Cheers,

Zhongze Liu

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel