[Xen-devel] [OSSTEST PATCH 2/4] README: Better documentation of recipes, db, etc.

[Ian Jackson <ian.jackson@xxxxxxxxxxxxx>]

CC: George Dunlap <george.dunlap@xxxxxxxxxx>
Signed-off-by: Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>
---
 README | 75 ++++++++++++++++++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 60 insertions(+), 15 deletions(-)

diff --git a/README b/README
index ffe0018..93129e3 100644
--- a/README
+++ b/README
@@ -8,32 +8,55 @@ Terminology
 
 "flight":
 
-    Each run of osstest is referred to as a "flight". Each flight is
-    given a unique ID (a number or name).
+    Each run of osstest is referred to as a "flight".  Each flight is
+    given a unique ID.
+
+    Standalone mode generally uses a flight named "standalone", which
+    is frequently erased and reused.  In "Executive" mode (used for
+    production osstest instances) flights are numbered and the
+    metadata is permanently recorded.
 
 "job":
 
-    Each flight consists of one or more "jobs". These are a sequence
+    Each flight consists of one or more "jobs".  These are a sequence
     of test steps run in order and correspond to a column in the test
-    report grid. They have names like "build-amd64" or
-    "test-amd64-amd64-pv". A job can depend on the output of another
+    report grid.  They have names like "build-amd64" or
+    "test-amd64-amd64-pv".  A job can depend on the output of another
     job in the flight -- e.g. most test-* jobs depend on one or more
     build-* jobs.
 
+    A job has a named "recipe", which indicates what things need to be
+    done in what order: generally, a sequence of ts-* scripts.  The
+    recipes are defined by their implementations in sg-run-job.
+
+    The set of jobs for any particular kind of flight is defined in
+    make-flight, make-*-flight, and mfi-*.
+
 "step":
 
-    Each job consists of multiple "steps" which is an individual test
-    operation, such as "build the hypervisor", "install a guest",
-    "start a guest", "migrate a guest", etc. A step corresponds to a
-    cell in the results grid. A given step can be reused in multiple
-    different jobs, e.g. the "xen build" step is used in several
-    different build-* jobs. This reuse can be seen in the rows of the
-    results grid.
+    Running a job consists of running its individual "steps".  Each
+    step is an individual test operation, such as "build the
+    hypervisor", "install a guest", "start a guest", "migrate a
+    guest", etc.
+
+    Generally a step corresponds to one execution of some ts-* script.
+
+    steps have a "testid" which is used to uniquely identify the same
+    operation in different test runs.  This is used for identifying
+    regressions, bisection, and so on.
+
+    The steps for a job, including the testids, are defined by the
+    recipe in sg-run-job (see "job", above).
+
+    Each step run in a job becomes a cell in the HTML results grid.  A
+    given step might appear in multiple different jobs, e.g. the "xen
+    build" step is used in several different build-* jobs.  This can
+    be seen in the rows of the results grid.
 
 "runvar":
 
-    A runvar is a named textual variable associated with each job in a
-    given flight. They serve as both the inputs and outputs to the
+    A runvar is a named textual variable associated with a given job
+    and flight. They serve as both the inputs and outputs to the
     job.
 
     For example a Xen build job may have input runvars "tree_xen" (the
@@ -55,6 +78,15 @@ Terminology
         * the parameters of the guest to test (e.g. distro, PV vs HVM
           etc).
 
+    Runvar names often have structure, being assembled out of various
+    pieces; and the names are sometimes parsed, too.
+
+flights, jobs and runvars are kept in the database (in standalone
+mode, "standalone.db").  As for steps, only steps which have been
+started are recorded in the db; steps which have yet to be attempted
+are not in the database.  Also, if a ts-* script is run by hand, this
+is not recorded as a step.
+
 Operation
 =========
 
@@ -73,12 +105,25 @@ referenced by each job's configuration. It then runs each 
of these in
 turn, taking into account the prerequisites etc, by calling the
 relevant "ts-*" scripts.
 
+Each ts-* is a convenient unit of test execution and reporting.  Most
+ts-* scripts are written in perl, although simple shell wrappers are
+sometimes used.  ts-* scripts expect OSSTEST_FLIGHT and OSSTEST_JOB to
+be set in the environment.  Most of them also expect to be told extra
+information on the command line - notably, which host(s) to operate
+on.
+
+In automatic operation, the command line arguments for each test
+step's ts-* script invocation are fixed by the recipe defined in
+sg-run-job.
+
 When running in standalone mode it is possible to run any of these
 steps by hand, ("mg-execute-flight", "sg-run-job", "ts-*") although
 you will need to find the correct inputs (some of which are documented
 below) and perhaps take care of prerequisites yourself (e.g. running
 "./sg-run-job test-armhf-armhf-xl" means you must have done
-"./sg-runjob build-armhf" and "build-armhf-pvops" first.
+"./sg-runjob build-armhf" and "build-armhf-pvops" first.  When running
+a ts-* script manually one often wants to specify "host=<hostname>".
+(See the head comment for "selecthost" in Osstest/TestSupport.pm.)
 
 Results
 =======
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel