[Xen-devel] Re: Domgrps/SchedGrps Merge RFC: domgrps-vmm

[Chris <hap10@xxxxxxxxxxxxxx>]

On Feb 11, 2008, at 5:07 PM, Mike D. Day wrote:
> > diff -urN xen-unstable/xen/arch/powerpc/powerpc64/ 
> > hypercall_table.S xen-unstable-domgrps/xen/arch/powerpc/powerpc64/ 
> > hypercall_table.S
> > --- xen-unstable/xen/arch/powerpc/powerpc64/hypercall_table.S	 
> > 2007-08-06 17:59:54.000000000 -0400
> > +++ xen-unstable-domgrps/xen/arch/powerpc/powerpc64/ 
> > hypercall_table.S	2007-11-19 18:42:00.000000000 -0500
> > @@ -41,6 +41,7 @@
> >         .quad 0 /* do_hvm_op */
> >         .quad do_sysctl             /* 35 */
> >         .quad do_domctl
> > +        .quad do_domgrpctl
> >         .rept NR_hypercalls-((.-__hypercall_table)/8)
> >         .quad do_ni_hypercall
> >         .endr
> Rather than creating a new hypercall, it may be best to implement all
> the grouping and grouping control as part of a sub-command of the
> domctrl hypercall. Then we don't need to add this extra hypercall into
> the table of every platform.
Ack.  There would be no functionality or flexibility lost by making  
group operations part of the domctl hypercall and doing so would also  
shave off a few LOC.  The reasons I had for make domgrpctl operations  
a separate hypercall are now moot.
> > +
> > +    if (d->group)
> > +        grp = d->group;
> > +    else
> > +        grp = find_grp_by_id(NULL_GROUP_ID);
> > +
> Wondering why we have to search for the null group. Why can't the null
> group be statically initialized and then always referred to by a
> pointer?
Ack.  The searching for the null group is an artifact of an older  
design.
> In fact, what is the role of the null group? I see that it is
> a real domain group and it is created automatically. Also it can't be
> created by a hypercall.
The purpose of the null group is to provide a means of efficiently  
addressing all domains that aren't explicitly in a group.  Although  
it's a real group, it isn't intended to be used as one from a user  
perspective.  Perhaps it would be useful for xm to display domains in  
the null group as "ungrouped."
> All domains that do not belong to a group actually belong to the null
> group. Could it also be just as easy for each domain to have an
> identity group (itself), to which it belongs in the absence of any
> other group membership? Or is there a specific purpose to the null
> group?
The current null group implementation isn't the only way to provide  
an efficient means to identify ungrouped domains, so I'm open to  
alternatives.  That said, is there a win to automatically creating a  
group for each domain?  When needed, a group can be created for an  
existing domain.
The null group should not provide any privileges or otherwise induce  
relationships between members.  The null group was intended primarily  
as a convenient tool for security policy analysis where it's useful  
to ensure the same property(ies) apply to all ungrouped domains.
> > +#define SERIALIZE_LINKED_LIST(op_name, list_name)              \
> > +    rcu_read_lock(&grp->op_name##_read_lock);                   \
> > +    memset(&info->op_name, 0, sizeof(domid_t)*MAX_GROUP_SIZE);  \
> > +    if (!list_empty(&grp->op_name)) {                           \
> > +        i = 0;                                                 \
> > +        list_for_each_entry(dom, &grp->op_name, list_name) {    \
> > +            info->op_name[i] = dom->domain_id;                   \
> > +                   i++;                                                \
> > +       }                                                       \
> > +    } else                                                     \
> > +       info->op_name[i] = NULL_GROUP_ID;                    \
> > +    rcu_read_unlock(&grp->op_name##_read_lock);
> This should really be cleaned up and made into an inline function. It
> also refers to a variable declared outside of the macro and parameters
> (info). It's not clear what's happening inside the macro.
Ack.

> > +#define RM_DOM_FROM_LIST(list_name, entry)     \
> > +    spin_lock(&grp->list_name##_update_lock);   \
> > +    if (!list_empty(&grp->list_name))           \
> > +        list_del(&dom->entry);                  \
> > +    spin_unlock(&grp->list_name##_update_lock);
> > +
> Another good candidate for inlining.
Ack.

> > +
> > +       /* skip it if dom is already a member */
> > +       if (dom_in_member_list(dom->domain_id, grp))
> > +               goto out;
> > +       
> > +       /* remove dom from old group */
> > +       if (dom->group != NULL)
> > +               del_dom_from_grp(dom);
> So a domain can only belong to one group at a time. Would it ever be
> useful for a domain to belong to more than one group? This type of
> restriction seems to work against the idea of a general grouping
> concept. For example, all domains that belong to a scheduling group
> (assuming we eventually merge domgrp and schedgrp) would automatically
> be removed from a scheduling group if added to any other type of
> group.
I agree with Samuel; it makes sense that a domain would be removed  
from its schedgrp when changing its domgrp.
In addition to the aesthetic issues Samuel mentioned (domains  
appearing in multiple listings), domgrps intentionally disallows  
membership in multiple groups because of the complications it would  
create.  Consider the following example with 3 domains and 2 groups.   
DomA and domB are in grpX; domB and domC are in grpY.  In this  
scenario XSM security policy expresses two properties: (1) group  
members can communicate with each other and (2) there is no cross- 
group communication.  However, from an information flow perspective  
grpX and grpY are effectively a single group because domB is a member  
of both and can be used to pass information between groups.  While it  
is possible to detect this policy conflict, doing so loses the  
simplicity gains of integrating groups with security policy.
In short, allowing domains to join multiple groups creates many more  
problems than it solves and there are extremely few good use cases  
(if any) where it's a win.  Contrary to popular belief, my goal is to  
add as little code to Xen as possible.  :)
> This argues for keeping different types of groups under different
> grouping infrastructures unless we can figure out a way for a domain
> to have multiple group memberships. It may be too complicated to do  
> so.
To increase flexibility, I propose allowing domains to opt in to/out  
of schedgrps (or any other feature that is integrated with the domgrp  
mechanism).  Can you describe a likely scenario where this extra  
ability doesn't meet your needs?
> > +int pause_grp(dgid_t dgid)
> > +{
> > +       int ret = 0;
> > +       struct domain *member;
> > +       struct domain_group *grp;
> > +
> > +       if (dgid == 0) {
> > +               ret = -EPERM;
> > +               goto out;
> > +       }
> > +
> > +       grp = find_grp_by_id(dgid);
> > +
> > +       if (grp == NULL) {
> > +               ret = -ESRCH;
> > +               goto out;
> > +       }
> > +
> > +       spin_lock_recursive(&grp->grp_big_lock);
> > +       rcu_read_lock(&grp->member_list_read_lock);
> > +       /* could ignore interupts during this loop to increase atomicity */
> > +       list_for_each_entry(member, &grp->member_list, member) {
> > +               if (member != current->domain && member->domain_id != 0)
> > +                       domain_pause_by_systemcontroller(member);
> > +       }
> > +       rcu_read_unlock(&grp->member_list_read_lock);
> > +       spin_unlock_recursive(&grp->grp_big_lock);
> > +      out:
> > +       return ret;
> > +}
> > +
> Groups are paused in the order in which they are added to the group,
> right? (FIFO). What do we do when we have groups which are dependant
> upon each other in certain ways.
In an earlier domgrps prototype I had a means for expressing  
dependencies that was read in from a group configuration file and was  
propagated into the VMM.  This allowed fine-grained dependencies to  
be expressed and handled at the VMM level.  However, I decided to  
remove it from the VMM because of the complexity it added.  Although  
there are certainly advantages to having that information in the VMM,  
my opinion today is that that functionality belongs in the control  
plane.
> For example, the stub domain and HVM domain. At first glance, it seems
> that you would want to pause the hvm domain before pausing its stub
> domain. And then the reverse on resume: unpause the stub domain, then
> unpause the hvm domain. (I could have it backward). But the point is,
> group operations may have inter-domain dependencies that are not
> accounted for simply by the order in which the domains are added to a
> group.  This is true for pause, unpause, start, migrate, etc., on down
> the line of possible group operations.
I agree.  The configuration-file based approach I mentioned earlier  
allowed users to specify reactions to various events and error  
conditions to include the events you listed above.  That  
functionality is not submitted as part of this patch, but it  
certainly has a place in future work.
I greatly appreciate the feedback.

-Chris

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel