Re: [Xen-devel] Proposal to allow setting up shared memory areas between VMs from xl config file

[Jarvis Roach <Jarvis.Roach@xxxxxxxxxxxxxxx>]

> -----Original Message-----
> From: Stefano Stabellini [mailto:sstabellini@xxxxxxxxxx]
> Sent: Friday, May 19, 2017 1:47 PM
> To: Zhongze Liu <blackskygg@xxxxxxxxx>
> Cc: Wei Liu <wei.liu2@xxxxxxxxxx>; Stefano Stabellini
> <sstabellini@xxxxxxxxxx>; xen-devel@xxxxxxxxxxxxxxxxxxxx; Julien Grall
> <julien.grall@xxxxxxx>; Ian Jackson <ian.jackson@xxxxxxxxxxxxx>; Jarvis
> Roach <Jarvis.Roach@xxxxxxxxxxxxxxx>; edgari@xxxxxxxxxx
> Subject: Re: Proposal to allow setting up shared memory areas between VMs
> from xl config file
> 
> On Sat, 20 May 2017, Zhongze Liu wrote:
> > Hi, Wei,
> >
> > >>>>2017-05-19 17:33 GMT+08:00 Wei Liu <wei.liu2@xxxxxxxxxx>:
> > > On Thu, May 18, 2017 at 11:09:40AM -0700, Stefano Stabellini wrote:
> > >> > >
> > >> > > Yes, those attributes are necessary and should be explicitly
> > >> > > specified in the config file. I'll add them in the next version
> > >> > > of this proposal. And taking the granularity into
> > >> > > consideration, what do you say if we change the entries into
> something like:
> > >> > > 'start=0xcafebabe, end=0xdeedbeef, granularity=4K, prot=RWX'.
> > >> >
> > >> > I realised I may have gone too far after reading your reply.
> > >> >
> > >> > What is the end purpose of this project? If you only want to
> > >> > insert a mfn into guest address space and don't care how the
> > >> > guest is going to map it, you can omit the prot= part. If you
> > >> > want stricter control, you will need them -- and that would also
> > >> > have implications on the hypervisor code you need.
> > >> >
> > >> > I suggest you write the manual for the new mechanism you propose
> first.
> > >> > That way you describe the feature in a sysadmin-friendly way.
> > >> > Describe the syntax, the effect of the new mechanism and how
> > >> > people are supposed to use it under what circumstances.
> > >>
> > >> The memory sharing mechanism should enable guests to communicate
> > >> with each other using a shared ring. That implies that the memory
> > >> needs to be read-write, but I can imagine there are use cases for
> > >> it to be read-only too. I think it is a good idea to specify it.

We've been seeing increased need for controls over other memory attributes 
(cacheability, shareability) in addition to access.


> > >> However, I do not think we should ask Zhongze to write a protocol
> > >> specification for how these guests should communicate. That is out
> > >> of scope.
> > >
> > > That's right. This is out of scope. I didn't mean to ask Zhongze to
> > > write a protocol specification.
> > >
> >
> > Yes, describing the syntax and the effect of the new mechanism, as
> > well as example usecases, will be of great importance. I'm now waiting
> > for Stefano to get some real llfe customer usecases.
> 
> [...]
> 
> > > There is a certain level of trust in the frontend / backend model.
> > > The frontend needs to trust backend to a certain degree. A user
> > > knows what to expect or do if one side misbehaves.
> > >
> > > But the way this proposal is phrased is that this is to construct a
> > > communication channel, i.e. it reads to me from a user's perspective
> > > you don't give one guest more trust than the other. This needs clarifying.
> > >
> >
> > IMHO, Since the shared memory is statically specified in the xl config
> > files, VMs don't have to ability to dynamically ask for a shared page
> > from other VMS through this new mechanism,  The xl file is written by
> > the sys admin, so I think it's the administrators'
> > duty to make it right.
> 

I think so too. From my embedded aviation experience, instead of a "sys admin" 
you have a "system integrator", an entity responsible for configuring the 
system correctly. For Xen-ARM, I'd see that covering the system device-tree and 
guest .cfg (including any partial DTs).

> While waiting for Jarvis and Edgar to provide more befitting information, I'll
> try to fill in myself. There is a demand to run "bare-metal" applications on
> embedded boards (such as the new Xilinx Zynq MPSoC board). People in
> those communities actually call them "bare-metal". They are written in C and
> run directly in kernel mode.
> There is no operating system or unikernel.  They can run bare-metal or inside
> a Xen VM. Usually they drive one or more devices (or FPGAs) with extremely
> low latency and overhead. An example of such an application is tbm (which is
> the one I used to do irq latency measurements):
> 
> https://github.com/edgarigl/tbm/app/xen/guest_irq_latency/apu.c
> 
> Let's suppose they you have one bare-metal app to drive a set of FPGA space
> and another one to drive another set of FPGA space (or two different
> devices), you might need the two apps to exchange information.
> 
> The owner knows exactly how many apps she is going to have on the board
> and who needs to communicate with whom from the start. It is a fully static
> configuration.
> 
> In this scenario, she is going to write to the VM config files of the two apps
> that one page will be shared among the two, so that they can send each
> other messages. She will hard-code the address of the shared page in her
> "bare-metal" app.
> 
> There is no frontend and backend (as in the usual Xen meaning). In some
> configurations one app might be more critical than the other, but in some
> other scenarios they might have the same criticality.
> 
> If, as Jan pointed out, we need to call out explicitly which is the frontend 
> and
> which is the backend for page ownership reasons, then I suggested we
> expose that configuration to the user, so that she can choose.

Thank you Stefano, that is a good representation of the need. 

I am not familiar with what all page ownership entails, if someone knows of a 
good link please send to me privately so I can educate myself. :)

Because of my ignorance the following may fly in the face of some design 
paradigm I'm not aware of, but I would expect to see these shared memory 
regions treated as a system resource, analogous to physical I/O periphs, 
defined so that Xen and Dom0 don't try to use them, and allocated to guests as 
part of a static configuration under control of a system integrator. Something 
would be needed in the system DT  to "carve" these regions out of the rest of 
memory, and then something in the guest's .cfg that gives guest access to the 
location. We can already do that, albeit with some limitations, using the 
"iomem" attribute in the .cfg and "uio" nodes in the system device tree. I 
mentioned this elsewhere, but if we just had greater control over the memory 
attributes via the "iomem" attribute in the .cfg I think we'd have most of the 
functionality we'd need.



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel