Re: [Xen-devel] [RFC v2]Proposal to allow setting up shared memory areas between VMs from xl config file

[Zhongze Liu <blackskygg@xxxxxxxxx>]

Hi Wei,

Thank you for your valuable comments.

2017-06-21 23:09 GMT+08:00 Wei Liu <wei.liu2@xxxxxxxxxx>:
> On Wed, Jun 21, 2017 at 01:18:38AM +0800, Zhongze Liu wrote:
>> ====================================================
>> 1. Motivation and Description
>> ====================================================
>> Virtual machines use grant table hypercalls to setup a share page for
>> inter-VMs communications. These hypercalls are used by all PV
>> protocols today. However, very simple guests, such as baremetal
>> applications, might not have the infrastructure to handle the grant table.
>> This project is about setting up several shared memory areas for inter-VMs
>> communications directly from the VM config file.
>> So that the guest kernel doesn't have to have grant table support (in the
>> embedded space, this is not unusual) to be able to communicate with
>> other guests.
>>
>> ====================================================
>> 2. Implementation Plan:
>> ====================================================
>>
>> ======================================
>> 2.1 Introduce a new VM config option in xl:
>> ======================================
>> The shared areas should be shareable among several (>=2) VMs, so
>> every shared physical memory area is assigned to a set of VMs.
>> Therefore, a “token” or “identifier” should be used here to uniquely
>> identify a backing memory area.
>>
>> The backing area would be taken from one domain, which we will regard
>> as the "master domain", and this domain should be created prior to any
>> other "slave domain"s. Again, we have to use some kind of tag to tell who
>> is the "master domain".
>>
>> And the ability to specify the attributes of the pages (say, WO/RO/X)
>> to be shared should be also given to the user. For the master domain,
>> these attributes often describes the maximum permission allowed for the
>> shared pages, and for the slave domains, these attributes are often used
>> to describe with what permissions this area will be mapped.
>> This information should also be specified in the xl config entry.
>>
>
> I don't quite get the attribute settings. If you only insert a backing
> page into guest physical address space with XENMEM hypercall, how do you
> audit the attributes when the guest tries to map the page?
>

I'm still considering about this, and any suggestions are welcomed.
The current plan
I have in mind is XENMEM_access_op.

>> To handle all these, I would suggest using an unsigned integer to serve as 
>> the
>> identifier, and using a "master" tag in the master domain's xl config entry
>> to announce that she will provide the backing memory pages. A separate
>> entry would be used to describe the attributes of the shared memory area, of
>> the form "prot=RW".
>
> I think using an integer is too limiting. You would need the user to
> know if a particular number is already used. Maybe using a number is
> good enough for the use case you have in mind, but it is not future
> proof. I don't know how sophisticated we want this to be, though.
>

Sounds reasonable. I chose integers because I think integers are fast
and easy to
manipulate. But integers are somewhat hard to memorize and this isn't
a good thing
from a user's point of view. So maybe I'll make it a string with a
maximum size of 32
or longer.

>> For example:
>>
>> In xl config file of vm1:
>>
>>     static_shared_mem = ["id = ID1, begin = gmfn1, end = gmfn2,
>>                           granularity = 4k, prot = RO, master”,
>>                          "id = ID2, begin = gmfn3, end = gmfn4,
>
> I think you mean "gpfn" here and below.
>

Yes, according to https://wiki.xenproject.org/wiki/XenTerminology, the section
"Address Spaces", gmfn == gpfn for auto-translated guests. But this usage
seems to be outdated and should be phased out according to include/xen/mm.h.
And just as what Julien has pointed out, the term "gfn" should be used here.

>>  granularity = 4k, prot = RW, master”]
>>
>> In xl config file of vm2:
>>
>>     static_shared_mem = ["id = ID1, begin = gmfn5, end = gmfn6,
>>                           granularity = 4k, prot = RO”]
>>
>> In xl config file of vm3:
>>
>>     static_shared_mem = ["id = ID2, begin = gmfn7, end = gmfn8,
>>                           granularity = 4k, prot = RW”]
>>
>> gmfn's above are all hex of the form "0x20000".
>>
>> In the example above. A memory area ID1 will be shared between vm1 and vm2.
>> This area will be taken from vm1 and mapped into vm2's stage-2 page table.
>> The parameter "prot=RO" means that this memory area are offered with 
>> read-only
>> permission. vm1 can access this area using gmfn1~gmfn2, and vm2 using
>> gmfn5~gmfn6.
>> Likewise, a memory area ID will be shared between vm1 and vm3 with read and
>> write permissions. vm1 is the master and vm2 the slave. vm1 can access the
>> area using gmfn3~gmfn4 and vm3 using gmfn7~gmfn8.
>>
>> The "granularity" is optional in the slaves' config entries. But if it's
>> presented in the slaves' config entry, it has to be the same with its 
>> master's.
>> Besides, the size of the gmfn range must also match. And overlapping backing
>> memory areas are well defined.
>>
>
> What do you mean by "well defined"?

Em...I think I should have put it in a more clear way. In fact, I mean
that overlapping
areas are allowed, and when two areas overlap with each other, any
operations done
on the overlapping area will be seen on both sides. Besides this, they
just act like two
independent areas. And the job of serializing the access to the
overlapping area is
left to the user.

>
> Why is inserting a sub-range not allowed?
>

This is also a feature under consideration.Maybe the use cases that I have
in mind is not that complicated, so I chose to keep it simple. But
after giving it
a second thought, I found this will not add too much complexity to the code and
will be useful in some cases. So I think I'll allow this in my next
version of the proposal.

>> Note that the "master" tag in vm1 for both ID1 and ID2 indicates that vm1
>> should be created prior to both vm2 and vm3, for they both rely on the pages
>> backed by vm1. If one tries to create vm2 or vm3 prior to vm1, she will get
>> an error. And in vm1's config file, the "prot=RO" parameter of ID1 indicates
>> that if one tries to share this page with vm1 with, say, "WR" permission,
>> she will get an error, too.
>>

Cheers,

Zhongze Liu

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel