Re: [Xen-devel] Proposal to allow setting up shared memory areas between VMs from xl config file

[Wei Liu <wei.liu2@xxxxxxxxxx>]

On Thu, May 18, 2017 at 11:09:40AM -0700, Stefano Stabellini wrote:
> > > 
> > > Yes, those attributes are necessary and should be explicitly specified in 
> > > the
> > > config file. I'll add them in the next version of this proposal. And 
> > > taking the
> > > granularity into consideration, what do you say if we change the entries 
> > > into
> > > something like:
> > > 'start=0xcafebabe, end=0xdeedbeef, granularity=4K, prot=RWX'.
> > 
> > I realised I may have gone too far after reading your reply.
> > 
> > What is the end purpose of this project? If you only want to insert a
> > mfn into guest address space and don't care how the guest is going to
> > map it, you can omit the prot= part. If you want stricter control, you
> > will need them -- and that would also have implications on the
> > hypervisor code you need.
> > 
> > I suggest you write the manual for the new mechanism you propose first.
> > That way you describe the feature in a sysadmin-friendly way.  Describe
> > the syntax, the effect of the new mechanism and how people are supposed
> > to use it under what circumstances.
> 
> The memory sharing mechanism should enable guests to communicate with
> each other using a shared ring. That implies that the memory needs to be
> read-write, but I can imagine there are use cases for it to be read-only
> too. I think it is a good idea to specify it.
> 
> However, I do not think we should ask Zhongze to write a protocol
> specification for how these guests should communicate. That is out of
> scope.

That's right. This is out of scope. I didn't mean to ask Zhongze to
write a protocol specification.

> 
> 
> > > >> In the example above. A memory area A1 will be shared between
> > > >> vm1 and vm2 -- vm1 can access this area using addr_range1
> > > >> and vm2 using addr_range3. Likewise, a memory area A2 will be
> > > >> shared between vm1 and vm3 -- vm1 can access A2 using addr_range2
> > > >> and vm3 using addr_range4.
> > > >>
> > > >> The shared memory area denoted by an identifier IDx will be
> > > >> allocated when it first appear, and the memory pages will be taken from
> > > >> the first VM whose static_shared_mem list contains IDx. Take the above
> > > >> config files for example, if we instantiate vm1, vm2 and vm3, one after
> > > >> another, the memory areas denoted by ID1 and ID2 will both be allocated
> > > >> in and taken from vm1.
> > > >
> > > > Hmm... I can see some potential hazards. Currently, multiple xl 
> > > > processes
> > > > are serialized by a lock, and your assumption is the creation is done in
> > > > order, but suppose sometime later they can run in parallel. When you
> > > > have several "xl create" and they race with each other, what will
> > > > happen?
> > > >
> > > > This can be solved by serializing in libxl or hypervisor, I think.
> > > > It is up to you to choose where to do it.
> > > >
> > > > Also, please consider what happens when you destroy the owner domain
> > > > before the rest. Proper reference counting should be done in the
> > > > hypervisor.
> > > >
> > > 
> > > Yes, the access to xenstore and other shared data should be serialized
> > > using some kind of lock.
> > > 
> > > >
> > > >>
> > > >> ======================================
> > > >> 2.2 Store the mem-sharing information in xenstore
> > > >> ======================================
> > > >> This information should include the length and owner of the area. And
> > > >> it should also include information about where the backing memory areas
> > > >> are mapped in every VM that are using it. This information should be
> > > >> known to the xl command and all domains, so we utilize xenstore to keep
> > > >> this information. A current plan is to place the information under
> > > >> /local/shared_mem/ID. Still take the above config files as an example:
> > > >>
> > > >> If we instantiate vm1, vm2 and vm3, one after another,
> > > >> “xenstore ls -f” should output something like this:
> > > >>
> > > >>
> > > >> After VM1 was instantiated, the output of “xenstore ls -f”
> > > >> will be something like this:
> > > >>
> > > >>     /local/shared_mem/ID1/owner = dom_id_of_vm1
> > > >>
> > > >>     /local/shared_mem/ID1/size = sizeof_addr_range1
> > > >>
> > > >>     /local/shared_mem/ID1/mappings/dom_id_of_vm1 = addr_range1
> > > >>
> > > >>
> > > >>     /local/shared_mem/ID2/owner = dom_id_of_vm1
> > > >>
> > > >>     /local/shared_mem/ID2/size = sizeof_addr_range1
> > > >>
> > > >>     /local/shared_mem/ID2/mappings/dom_id_of_vm1 = addr_range2
> > > >>
> > > >>
> > > >> After VM2 was instantiated, the following new lines will appear:
> > > >>
> > > >>     /local/shared_mem/ID1/mappings/dom_id_of_vm2 = addr_range3
> > > >>
> > > >>
> > > >> After VM2 was instantiated, the following new lines will appear:
> > > >>
> > > >>     /local/shared_mem/ID2/mappings/dom_id_of_vm2 = addr_range4
> > > >>
> > > >> When we encounter an id IDx during "xl create":
> > > >>
> > > >>   + If it’s not under /local/shared_mem, create the corresponding 
> > > >> entries
> > > >>      (owner, size, and mappings) in xenstore, and allocate the memory 
> > > >> from
> > > >>      the newly created domain.
> > > >>
> > > >>   + If it’s found under /local/shared_mem, map the pages to the newly
> > > >>       created domain, and add the current domain to
> > > >>       /local/shared_mem/IDx/mappings.
> > > >>
> > > >
> > > > Again, please think about destruction as well.
> > > >
> > > > At this point I think modelling after POSIX shared memory makes more
> > > > sense. That is, there isn't one "owner" for the memory. You get hold of
> > > > the shared memory via a key (ID in your case?).
> > > >
> > > 
> > > Actually, I've thought about the same question and have discussed this 
> > > with
> > > Julien and Stefano. And this what they told me:
> > > 
> > > Stefano wrote:
> > > "I think that in your scenario Xen (the hypervisor) wouldn't allow the
> > > first domain to be completely destroyed because it knows that its
> > > memory is still in use by something else in the system. The domain
> > > remains in a zombie state until the memory is not used anymore. We need
> > > to double-check this, but I don't think it will be a problem."
> > > 
> > 
> > This has security implications -- a rogue guest can prevent the
> > destruction of the owner.
> 
> We are going to use the same underlying hypervisor infrastructure, the
> end result should be no different than sharing memory via grant table
> from a security perspective. If not, then we need to fix Xen.
> 

There is a certain level of trust in the frontend / backend model. The
frontend needs to trust backend to a certain degree. A user knows what
to expect or do if one side misbehaves.

But the way this proposal is phrased is that this is to construct a
communication channel, i.e. it reads to me from a user's perspective you
don't give one guest more trust than the other. This needs clarifying.

> 
> > > and Julien wrote:
> > > "That's correct. A domain will not be destroyed until all the memory
> > > associated to it will be freed.
> > > A page will be considered free when all the reference on it will be
> > > removed. This means that if the domain who allocated the page die, it
> > > will not be fully destroyed until the page is not used by another
> > > domain.
> > > This is assuming that every domain using the page is taking a
> > > reference (similar to foreign mapping). Actually, I think we might be
> > > able to re-use the mapspace XENMAPSPACE_gmfn_foreign.
> > > Actually, I think we can re-use the same mechanism as foreign mapping (see
> > > Note that Xen on ARM (and x86?) does not take reference when mapping a
> > > page to a stage-2 page table (e.g the page table holding the
> > > translation between a guest physical address and host physical
> > > address)."
> > > 
> > > I've also thought about modeling after the POSIX way of sharing memory.
> > > If we do so, the owner of the shared pages should be Dom0, and we
> > > will have to do the reference counting ourselves, and free pages when 
> > > they're
> > > no longer needed. I'm not sure which method is better. What do you say?
> > > 
> > 
> > Assigning the page to Dom0 doesn't sound right to me either.
> > 
> > But the first step should really be defining the scope of the project.
> > Technical details will follow naturally.
> 
> I thought that Zhongze wrote it well in "Motivation and Description".
> What would you like to know in addition to that? 

A bit more details are needed. See above.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel