Re: [Xen-devel] [RFC][PATCH 0/5] Add V4V to Xen

[Tim Deegan <tim@xxxxxxx>]

Hi, 

Thanks for this.

Overall, it looks like Xen is doing a few things here:
 - nameservice for registering services and finding endpoints;
 - ring manipulation arithmetic;
 - copying data; and
 - notifying endpoints. 

The shared-ring logic was able to do all of these, with a few drawbacks:
 - The xenstore handshake stuff is really grotty;
 - grant maps can cause zombie domains; and
 - it doesn't do many-to-one multicast.

Is that a fair summary?

Using copy semantics intead of shared rings fixes the zombie domain
problem, and I think we should definitely take that.

We talked last week about ways we could improve xenstore to provde the
nameservice aspects of v4v; if that can be done I think it should be.

The rest of is within spitting distance of a multicall of grant copies
and event-channel pokes - except for the many-to-one multicast.
That relies on Xen policing the ring headers to make sure the length
fields are correct.

What's the use case for that?  Do you use it just to negotiate a
dedicated ring for each sender or is there a need for multiple data
streams to be multiplexed together?

I have a few questions on the design, inline below. 

At 10:42 +0100 on 07 Jun (1339065745), Jean Guyader wrote:
> v4v has a very simple architecture. The receiving domain registers a
> ring with xen.  The sending domain requests that xen inserts data into
> the receiving domain's ring.  Notification that there is new data to
> read is delivered by a VIRQ

Not an evtchn?

> , and a domain can request an interrupt be generated when sufficient
> space is available in another domain's ring. 

How is that arranged?  It doesn't look like the receiver notifies Xen
when it consumes a ring entry.

> The code that inserts
> the data into the ring is in xen, and xen writes a header describing
> the data and where it came from infront of the data. As both the ring
> manipulation and the header are written by the hypervisor, the
> receiving domain can trust their content and the format of the ring.

Does that save a copy on the receiving side?

> ** Addressing
> v4v_addr_t identifies an endpoint address.
>  typedef struct v4v_addr
>  {
>      uint32_t port;
>      domid_t domain;
>  } v4v_addr_t;
> struct v4v_ring_id uniquely identifies a ring on the platform.
> 
>  struct v4v_ring_id
>  {
>     struct v4v_addr addr;
>     domid_t partner;
>  };
> 
> When a send operation is performed, xen will attempt to find a ring 
> registered by destination domain, with the required port, and with
> a partner of the sending domain. Failing that it will then search
> for a ring with the correct port and destination domain, but with 
> partner set to V4V_DOMID_ANY.
> 
> ** Setup
> A domain first generates a ring, and a structure describing the pages
> in the ring.  Rings must be page aligned, and contiguous in virtual
> memory, but need not be contiguous in physical(pfn) or machine(mfn)
> memory. A ring is uniquely identified on the platform by its struct
> v4v_ring_id.
>
> A ring is contained in a v4v_ring_t
>  typedef struct v4v_ring
>  {
>     uint64_t magic;
>     /*
>      * Identifies ring_id - xen only looks at this during register/unregister
>      * and will fill in id.addr.domain
>      */
>     struct v4v_ring_id id;
>     /* length of ring[], must be a multiple of 16 */
>     uint32_t len;
>     /* rx_ptr - modified by domain */
>     uint32_t rx_ptr;
>     /* tx_ptr - modified by xen */
>     volatile uint32_t tx_ptr;
>     uint64_t reserved[4];
>     volatile uint8_t ring[0];
>  } v4v_ring_t;

(Digressing into code review for a minute, I don't think we should use
'volatile' in the public headers, and possibly it will be more efficient
to use barriers anyway if we're processing the received data in place).

> To register the ring, xen also needs a list of pfn that the ring occupies.
> This is provided in a v4v_pfn_list_t
>  typedef struct v4v_pfn_list_t
>  {
>     uint64_t magic;
>     uint32_t npage;
>     uint32_t pad;
>     uint64_t reserved[3];
>     v4v_pfn_t pages[0];
>  } v4v_pfn_list_t;
> 
> Registering the ring is done with a hypercall
> 
> int hypercall(NR_V4V,V4VOP_register_ring,v4v_ring_t *ring,v4v_pfn_list_t 
> *pfn_list);
> 
> On registering, xen will check that the tx_ptr and rx_ptr values are valid 
> (aligned
> and within the size of the ring) and modify them if they are not

Modify them?  Not return an error?

>, it will also update
> id.addr.domain with the calling domain's domid and take an internal copy of 
> all the
> relevant information. After registration the only field that xen will read is
> ring->rx_ptr, xen will write to ring->ring[] and ring->tx_ptr. 
> Thus pfn_list can be freed after the registration call. Critically, 
> registration is
> idempotent, and all a domain need do after hibernation or migration is 
> re-register all
> its rings. (NB the id.addr.domain field may change).
> 
> ** Sending
> 
> To send data a guest merely calls the send or sendv hypercall
> int hypercall(NR_V4V,V4VOP_send,v4v_addr_t *src,v4v_addr_t *dst,void *buf, 
> uint32_t len,
>               uint32_t protocol);

What does the source address do here?  Do I need to register a ring in
order to send to another ring?

Why the separate 'protocol' argument when 'port' is bound into the address
struct?  A single ring receives messages for a given port but all protocols? 

> the caller provides a source address, a destination address, a buffer, a 
> number of bytes to copy
> and a 32 bit protocol number. Xen ignores src->domain and instead uses the 
> domain id of the caller.
> 
> Xen will attempt to insert into a ring with id.addr equal to dst, and with
> id.partner equal to the caller's domid. Otherwise it will insert into a ring
> with id.addr equal to dst and id.partner equal to V4V_DOMID_ANY. If the 
> insert is successfull
> then Xen will trigger the V4V VIRQ line in the receiving domain.
> 
> ** Reception
> 
> Xen pads all messages on the ring to 16 bytes (the macro V4V_ROUNDUP is 
> provided 
> for convience), and inserts a message header infront of all messages it 
> copies onto the ring.
> 
>  struct v4v_ring_message_header
>  {
>     uint32_t len;
>     struct v4v_addr source;
>     uint16_t pad;
>     uint32_t protocol;
>     uint8_t data[0];
>  };
> 
> len is the number of bytes in the message including the struct 
> v4v_ring_message_header
> source specifies the source address from which the message came, 
> source.domain is
> set by xen, and source.port is taken from the arguments to the send or sendv 
> call.
> protocol is the protocol value given to the send call.
> 
> An  inline function is provided to make reading from the ring easier:
> 
>  ssize_t
>  v4v_copy_out (struct v4v_ring *r, struct v4v_addr *from, uint32_t * protocol,
>               void *_buf, size_t t, int consume)

I guess that answers my question about avoiding a copy on the receiver. :(

Cheers,

Tim.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel