Re: [Xen-devel] [RFC][PATCH 0/5] Add V4V to Xen

[Jean Guyader <jean.guyader@xxxxxxxxxxxxx>]

On 01/06 02:47, Ian Campbell wrote:
> On Thu, 2012-05-31 at 16:07 +0100, Jean Guyader wrote:
> > V4V is a copy based inter vm communication system.
> > 
> > Please have a look at this thread for more detail
> > about V4V.
> > http://lists.xen.org/archives/html/xen-devel/2012-05/msg01866.html
> > 
> > This patch series is work in progress but I wanted to
> > post it early enough so I can get feedback from people.
> 
> The main thing which is missing here, which makes it rather hard to
> review, is any kind of design documentation. It would also be great to
> see the rationale for why things have to be done this way.
> 
> For example it seems that there is a bunch of stuff being added to the
> hypervisor which could live in the context of some guest or service
> domain -- putting stuff like that in the hypervisor needs strong
> arguments and rationale why it has to be there rather than somewhere
> else.
> 
> Likewise perhaps the v4v hypercall could effectively be implemented with
> a multicall containing some grant copy ops and evtchn manipulations?
> 
> But in the absence of any descriptions of the whats, hows and whys of
> v4v its rather hard to have these sorts of conversations. The above are
> some concrete examples but I'm more interested in seeing the more
> general descriptions before we get into those.
> 

Here is some documentation of the Xen V4V API and the ring structure.
Let me know if you have other questions.

* Overview
** Architecture

v4v has a very simple architecture. The receiving domain registers a ring with 
xen. 
The sending domain requests that xen inserts data into the receiving domain's 
ring.
Notification that there is new data to read is delivered by a VIRQ, and a 
domain can
request an interrupt be generated when sufficient space is available in another 
domain's
ring. The code that inserts the data into the ring is in xen, and xen writes a 
header
describing the data and where it came from infront of the data. As both the 
ring manipulation
and the header are written by the hypervisor, the receiving domain can trust 
their content
and the format of the ring. 

** Addressing
v4v_addr_t identifies an endpoint address.
 typedef struct v4v_addr
 {
     uint32_t port;
     domid_t domain;
 } v4v_addr_t;
struct v4v_ring_id uniquely identifies a ring on the platform.

 struct v4v_ring_id
 {
    struct v4v_addr addr;
    domid_t partner;
 };

When a send operation is performed, xen will attempt to find a ring 
registered by destination domain, with the required port, and with
a partner of the sending domain. Failing that it will then search
for a ring with the correct port and destination domain, but with 
partner set to V4V_DOMID_ANY.

** Setup
A domain first generates a ring, and a structure describing the pages in the 
ring.
Rings must be page aligned, and contiguous in virtual memory, but need not be 
contiguous in physical(pfn) or machine(mfn) memory. A ring is uniquely 
identified
on the platform by its struct v4v_ring_id.

A ring is contained in a v4v_ring_t
 typedef struct v4v_ring
 {
    uint64_t magic;
    /*
     * Identifies ring_id - xen only looks at this during register/unregister
     * and will fill in id.addr.domain
     */
    struct v4v_ring_id id;
    /* length of ring[], must be a multiple of 16 */
    uint32_t len;
    /* rx_ptr - modified by domain */
    uint32_t rx_ptr;
    /* tx_ptr - modified by xen */
    volatile uint32_t tx_ptr;
    uint64_t reserved[4];
    volatile uint8_t ring[0];
 } v4v_ring_t;

To register the ring, xen also needs a list of pfn that the ring occupies.
This is provided in a v4v_pfn_list_t
 typedef struct v4v_pfn_list_t
 {
    uint64_t magic;
    uint32_t npage;
    uint32_t pad;
    uint64_t reserved[3];
    v4v_pfn_t pages[0];
 } v4v_pfn_list_t;

Registering the ring is done with a hypercall

int hypercall(NR_V4V,V4VOP_register_ring,v4v_ring_t *ring,v4v_pfn_list_t 
*pfn_list);

On registering, xen will check that the tx_ptr and rx_ptr values are valid 
(aligned
and within the size of the ring) and modify them if they are not, it will also 
update
id.addr.domain with the calling domain's domid and take an internal copy of all 
the
relevant information. After registration the only field that xen will read is
ring->rx_ptr, xen will write to ring->ring[] and ring->tx_ptr. 
Thus pfn_list can be freed after the registration call. Critically, 
registration is
idempotent, and all a domain need do after hibernation or migration is 
re-register all
its rings. (NB the id.addr.domain field may change).

** Sending

To send data a guest merely calls the send or sendv hypercall
int hypercall(NR_V4V,V4VOP_send,v4v_addr_t *src,v4v_addr_t *dst,void *buf, 
uint32_t len,
              uint32_t protocol);

the caller provides a source address, a destination address, a buffer, a number 
of bytes to copy
and a 32 bit protocol number. Xen ignores src->domain and instead uses the 
domain id of the caller.

Xen will attempt to insert into a ring with id.addr equal to dst, and with
id.partner equal to the caller's domid. Otherwise it will insert into a ring
with id.addr equal to dst and id.partner equal to V4V_DOMID_ANY. If the insert 
is successfull
then Xen will trigger the V4V VIRQ line in the receiving domain.

** Reception

Xen pads all messages on the ring to 16 bytes (the macro V4V_ROUNDUP is 
provided 
for convience), and inserts a message header infront of all messages it copies 
onto the ring.

 struct v4v_ring_message_header
 {
    uint32_t len;
    struct v4v_addr source;
    uint16_t pad;
    uint32_t protocol;
    uint8_t data[0];
 };

len is the number of bytes in the message including the struct 
v4v_ring_message_header
source specifies the source address from which the message came, source.domain 
is
set by xen, and source.port is taken from the arguments to the send or sendv 
call.
protocol is the protocol value given to the send call.

An  inline function is provided to make reading from the ring easier:

 ssize_t
 v4v_copy_out (struct v4v_ring *r, struct v4v_addr *from, uint32_t * protocol,
              void *_buf, size_t t, int consume)

v4v_copy_out will copy at most t bytes from the ring r and place them in buf, 
if it
is non-null. If from and protocol are not NULL pointers then the source address 
and
 protocol number will by copied into them. If consume is non-zero then the 
ring's
receive pointer will be advanced. The function returns the number of bytes of 
payload
that the message has (ie. the number of bytes passed to the original send or 
sendv call).
Thus, v4v_copy_out(r,NULL,NULL,NULL,0,0); will return the number of bytes in 
the next message,
and  v4v_copy_out(r,NULL,NULL,NULL,0,1);  will return the number of bytes in 
the next message
and delete it from the ring.

** Notification
If the receiving ring is full in a send or sendv hypercall, the hypercall will 
return with
the error -EAGAIN. When sufficient space becomes available in the ring xen will 
raise the V4V VIRQ 
line.


Jean

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel