Re: [Xen-devel] Testing for the Xen Project

[Lars Kurth <lars.kurth@xxxxxxx>]

Ian,

I think that is a good assessment and thanks for pointing out some gaps 
in my understanding. So to summarize:
* We have some local test suites which could be run, but are probably 
not as they are poorly documented
* We expect people to do some targetted local testing (presumably 
performed in a manual manner) of the features they developed and of 
those which may be impacted. Bt we don't actually always know whether 
they do.
* osstest (or system testing in general) is *extremely* valuable
* building out the infrastructure for system testing (aka number and 
diversity of boxes) would be extremely valuable - this really means 
funding hardware, hosting and sysadmin time
* It might also be worth considering spending some money kickstarting 
the actual tests (i.e. fleshing out the suites)
* *Test on demand* would be a nice long term goal
* Some members of the community are intending to make OSSTest more 
accessible by improving docs and sharing their experience
* At a minimum it ought to be possible to allow access to any employee 
of, a project member, since we have the opportunity through the 
membership, process to put whatever paperwork and agreements (acceptable 
use etc) in place.
Lars

On 09/12/2013 13:20, Ian Campbell wrote:
> On Tue, 2013-11-26 at 13:21 +0000, Lars Kurth wrote:
> > Hi all,
> >
> > you probably have all heard by now that the Xen Project Advisory Board
> > (a group of vendors who provide funds to the Xen Project that are
> > intended to be used for the good of the community) recently created the
> > Test Framework Working
> > Group.http://wiki.xenproject.org/wiki/AB_WG/Test_Frameworkcontains more
> > information about the group. The working group had its first meeting a
> > few weeks ago and one of the actions I had was to kick off a thread on
> > development lists to figure out what would help the developer community.
> >
> > I was planning to kick off this thread with some questions and options,
> > which reflect some discussions I had with individuals in the community,
> > various meetings (WG and AB meetings), etc. which I condensed into a
> > picture.
> >
> > This reflects my personal opinion (not a Citrix opinion) and is merely
> > intended to get a discussion going. Feel free to pick it apart: I wonât
> > be upset.
> >
> > First, I wanted to clear up a few misconceptions that I have heard from
> > a few people:
> >
> > * The Advisory Board has funds that can be used to create an
> > independently hosted test infrastructure to help the developer
> > community. However, funds are limited. Thus, it is important that we do
> > what is right for the Xen community in the short term and the longer
> > term. Otherwise, we will burn funds that could be used to help the Xen
> > community in other ways.
> > * The Test Framework Working Group is made up of people employed by
> > vendors who have some experience in testing.
> > * There is no intention to prescribe a test environment that you then
> > have to use. Advisory Board members made clear to me that they want to
> > make sure that what we end up with a solution that works for you.
> > * At the Xen Developer Summit two different solutions for system testing
> > were presented. The intention was to explain what is there and what we
> > can use going forward. A presentation on OSSTest which runs regularly
> > today was given. And one for XenRT, for which there is a plan to get a
> > small 3 box system up and running that can be used for you to look at.
> > Citrix volunteered to set this up at its own cost.
> > * Just to be clear: what works for you may be one of these, none of
> > these, both of them, â
> > * There may also be different answers in the short and the long run.
> > * At the end of the day, different community members will have different
> > views. Also the Advisory Board members who provide the funds, will also
> > have specific interests that they will push for. Thus, in all
> > likelihood, we will have to find a good enough compromise.
> > * The vast majority of Advisory Board members care about the Hypervisor
> > (and not so much about XAPI and Mirage OS). Thus, it is likely that the
> > focus of the test system would be the Hypervisor.
> >
> > So let me try and condense some of the arguments and opinions I heard
> > and information that is around. This list may be incomplete.
> >
> > == Work Flow ==
> >
> > I added this section, because some members of the community and the
> > working group had prior experience with attempts to introduce a test
> > infrastructure for an open source community in the past, and these may
> > not have worked as well as hoped. I made up some of the terminology.
> >
> > *Local testing*: the basic idea here is for a developer to write their
> [...]
>
> We have a small amount of local test suites in the tree (e.g. vif and
> disk config parsing have little test suites) but it could do with tying
> together with some infrastructure into something which is simple to run
> (currently it requires an installed Xen system and there is no one
> single way to run something).
>
> As you correctly suggest there is a limit to how much local testing can
> cover in terms of elapsed time, available resources, the configurations
> which can be reasonably set up, running on real hardware etc. IMHO This
> could benefit from an enthusiastic (or press-ganged by their
> manager ;-)) community member putting some time into tying it all
> together into something which we can ask people to run before submitting
> with a straight face.
>
>
> > *System testing*: both OSSTest and XenRT are essentially system test
> [...]
>
> I think most people use "system testing" to mean testing of the
> integrated whole, as opposed to e.g. unit testing. The current
> "automated test" which we have covers some aspects of both whole system
> and unit testing.
>
> Anyway, terminology aside, the existing osstest stuff is *extremely*
> valuable IMHO, and the system testing has been very useful over the
> majority of the lifetime of the xen project, at least as long as I've
> been involved. The main limitation is the amount of resources dedicated
> to it, in terms of hardware (and its location within citrix
> infrastructure doesn't help here) and test coverage.
>
> Even with its current set of tests and limited hardware it already tests
> far more than we could ever realistically ask people to do locally
> before submitting and it catches real issues on real hardware.
>
> Any local test stuff should obviously be integrated into the system
> tests as a step as well.
>
> I notice that your description of system test omits the targeted local
> testing which we expect contributors to do before submitting a patch --
> by targeted I mean you are changing $FOO therefore you should be trying
> $FOO! And if you think you might have an impact on $BAR you should be
> testing that too. I just mention it because your description seemed to
> imply (inadvertently I expect) that there was no testing at all between
> writing the code and the system tests running, which is not quite
> accurate.
>
> IMHO both local and system test are valuable. I think the local testing
> situation can be improved by people working within the community to do
> the work (in particular building out the infrastructure), whereas the
> system testing side of things would benefit greatly from any resourcing
> which the AB can provide in terms of hardware, hosting and sysadmin time
> etc. There is no doubt in my mind that this would be beneficial to the
> community in both the short and long term.
>
> It might also be worth considering spending some money kickstarting the
> actual tests (i.e. fleshing out the suites) in both cases, but I think
> ultimately I think this needs to be driven by community member (AB or
> otherwise) who care about particular functionality making sure the tests
> exist, probably by writing them. So in terms of budget I think that
> would be secondary to sorting out the hosting etc
>
> > *Test on demand:* this would be a mixture between local testing and
> [...]
>
> I think it would be nice long term goal to aim for this but short term
> the other two types of testing are more important.
>
> > IMHO, this would be a nice mid to long-term goal,
> > assuming it could be made to work with the funds we have.
> Heh, I should read right to the end ;-)
>
> > == OSSTest ==
> >
> > What runs now and thus easiest to get started on
> >
> > More Info
> > *http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/
> > *http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/
> > *http://www.youtube.com/watch?v=JxTFZIwZzJ8
> >
> > Problems:
> > * Runs on Citrix premises (thus general access is an issue)
> > * Ian Jackson is acting as sys-admin in his spare time. But, the
> > Advisory Board could provide resource to fix this
> > * Basic test coverage
> > * Not a lot of documentation right now (which is a bit of a barrier to
> > adoption)
> >
> > Risks
> > * Not well understood (maybe you guys can fill the gaps)
> This is slowly changing, Wei, Roger and myself have all done development
> with osstest and contributed (or are in the process of doing so) new
> bits of testing. I think Dario and Anthony have played with it too.
> There is certainly more which could be done here in terms of
> documentation. I at least was planning to make this part of my focus on
> future documentation and/or test days.
>
> I think all of the above applies equally to XenRT, either system is
> going to have a learning curve and is going to need documentation for
> the community etc.
>
> > == XenRT ==
> >
> > Used by Citrix for XenServer testing. Tarballs have been made available
> > by Citrix under a BSD license. But the code has not been put into live
> > repos: my understanding is that Citrix would do this, if the Xen
> > community believes this is valuable.
> >
> > More Info
> > *http://wiki.xenproject.org/wiki/Getting_Started_with_XenRT
> > *http://www.youtube.com/watch?v=s11_Iw7AI_U
> >
> > Problems:
> > * No publicly accessible demo instance (this is being worked on â to be
> > hosted on a small test bed at http://osuosl.org/ â work sponsored by Citrix)
> > * Currently does not yet support âxlâ (a âxlâ connector is being worked
> > on â sponsored by Citrix)
> > * Code not in yet public repo
> >
> > Potentially Interesting Properties:
> > * Very large test coverage (including performance, security and other
> > tests). Most of them should work once an âxlâ connector is in place
> I think that's rather optimistic. I would expect that a reasonable
> proportion of the interesting tests will require features of xapi to
> work, e.g. pools of hosts, storage management, networking etc and/or
> require some amount of reworking to function with xl.
>
> > * Been in production at scale for a long time: thus well understood
> > * XenRT has a lot of provisioning functionality and supports a
> > distributed architecture: aka the ability to manage machines in
> > different locations (data centres). The detail is abstracted away from
> > users. This creates some interesting possibilities. For example:
> > ** Hardware Vendors on the Advisory Board could provide hardware to the
> > community on their site (assuming that these can be hosted outside a
> > firewall). Some HW vendors on the AB indicated that this would indeed be
> > doable.
> > ** This would open up the opportunity to make available cutting edge or
> > âunusualâ HW for testing to the community.
> > ** It would also mean that machines that would be expensive to ship and
> > host by the project, could be hosted on premise by AB vendors
> > * XenRT has the capability to âinjectâ some test code on the fly (i.e.
> > the test code is attached to a job that is submitted).
> > * I checked this with the XenRT devs and the *Test on demand* approach
> > should be relatively easy to implement, but does not exist.
> >
> > I do not know what of the above would apply to OSSTest.
> I think it is all equally doable for either.
>
> > Risks
> > * Complexity
> > * The cost of supporting such a system may be too high
> > * Not in use by the community today
> > * Not clear whether a *local test* version of XenRT is feasible
> >
> > == Support and Ownership ==
> >
> > Whatever solution we go for, needs to be properly funded and looked
> > after.
>  From the remainder of the paragraph I think you are talking specifically
> about hiring a test person here I think?
>
> I think this is essential, the current testing is done on a shoe string
> and that is one of its main limiting factors.
>
> >   This is understood and the intention would be for the Xen Project
> > (aka Advisory Board) to fund a Linux Foundation employee to do this on
> > behalf of the Xen Project: this is a bit like Greg KH and others being
> > LF employees working on the kernel. Some vendors on the Advisory Board
> > indicated that providing Colo/hosting space and HW would be possible in
> > principle, which could help keeping the cost manageable.
> We should certainly be taking them up on those offers IMHO.
>
> > == Access ==
> >
> > Any central system, has of course the issue of access control and
> > managing users. This is obviously a barrier to entry (if we do not have
> > also a local test mechanism). Am wondering how other FOSS communities
> > handle this. This should certainly be the job of the Test Framework
> > owner (see above).
> At a minimum it ought to be possible to allow access to any employee of
> a project member, since we have the opportunity through the membership
> process to put whatever paperwork and agreements (acceptable use etc) in
> place.
>
> Unfettered access for anyone who rocks up and asks is a bit trickier.
> I'm quite happy to let that be the framework owner's problem ;-)
>
> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel