[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] [PATCH 00/27] Running benchmarks via OSSTest

Hello everyone,

This is a highly reworked and much more mature version of this old RFC series:


It is about integrating benchmarking capabilities into OSSTest. The series is a
lot bigger, because it is now capable of doing a lot more stuff. :-) All the
comments I got during review of the RFC have been addressed, of course... More
comments are of course welcome!

The series is available here:

  git://xenbits.xen.org/people/dariof/osstest.git benchmarking-with-osstest

So, about the patches. Patches 01 to 04, as well as patch 20, are preliminary
changes and fixes, necessary for the rest of the series to work properly, but
they are general enough to be considered separately from the series, if

Patches 05 to 12 make it possible to run an instance of the Unixbench (XXX)
benchmark via OSSTest. Management scripts, test scripts, jobs and recipes are
provided to download the benchmark, prepare the guest (PV or HVM), build the
benchmark in the guest, run it, and retrieve and process (include some super
fancy plots! :-D) the results.

To do so, do as follows (in standalone mode):

  $ ./standalone-reset -t bench
  $ ./mg-unixbench-download
  $ ./standalone run-job -h ultralisk 

One thing about patch 12. I tried, but I can't tell how I should invoke
./standalone in order to test the new parameter I'm adding... Can someone help
with this? IanC?

To have an idea of how the 'output' of one of this jobs looks like, have a look


Or here:


Patches 13 to 19 does the very same, but for Kernbench (XXX):

  $ ./mg-kernbench-download
  $ ./standalone run-job -h ultralisk 

Patches 21 to 27 aims at allowing running a benchmark both in a guest and on
baremetal, to investigate the performances loss due to the virtualization
overhead. To do that, for Unixbench and Kernbench, one just issue the following
commands (in standalone mode):

  $ ./standalone run-job -h ultralisk bench-hostcmp-unixbench-amd64-credit
  $ ./standalone run-job -h ultralisk bench-hostcmp-kernbench-amd64-credit2

To have an idea of how the 'output' of one of these jobs looks like, have a
look here:


 * in patch 02/27, I haven't actually checked whether also grub1 and/or uboot
   needs fixing
 * When comparing performance, I think it would be best to always use the same
   kernel, in host and guests. This happens by default for PV guest, while it
   does not for HVM guests. This of course can be solved pretty easily, and I
   plan to do it either in future posting of this series, or with a follow-up

Dario Faggioli (27):
      ts-devbian-hvm-install: prune "cdrom:" from install sources
      Osstest/Debian.pm: fix identifying a Linux baremetal grub2 entry
      Guest setup: allow the amount of RAM to be a runvar
      Osstest/TestSupport.pm: Introduce target_getfile_[root_]stash()
      mg-unixbench-download: new script for downloading the unixbench archive
      ts-unixbench-build: prep the environment for running unixbench
      ts-unixbench-run: kick off the benchmark on the target
      ts-unixbench-reslts: for retrieving the results
      ts-unixbench-reslts: process and plot bench results
      sg-run-job: recipes for the unixbench jobs
      make-bench-flight: to create a benchmarking flight
      standalone-reset: introduce a new -t option
      mg-kernbench-download: new script for downloading kernbench
      ts-kernbench-build: prep the environment for running kernbench
      ts-kernbench-run: kick off the benchmark on the target
      ts-unixbench-reslts: retrieve and stash kernbench results
      ts-kernbench-reslts: process and plot bench results
      sg-run-job: recipes for the kernbench jobs
      make-bench-flight: create kernbench jobs
      Osstest/TestSupport.pm: read hosts' hardware characteristics
      ts-bench-hostcmp-guest-prep: new script
      ts-bench-hostcmp-host-prep: new script
      ts-bench-hostcmp-host-reset: new script
      Recipes and jobs for running unixbench both on host and guest
      ts-bench-hostcmp-post: add plotting facilities
      Kernbench perf comparison between host and guest
      ts-bench-hostcmp-post: add plotting facilities

 Osstest/Benchmarking.pm     |  210 +++++++++++++++++++++++++++++++++++++++++++
 Osstest/Debian.pm           |   54 +++++++----
 Osstest/TestSupport.pm      |  166 +++++++++++++++++++++++++++++++++-
 README                      |   36 +++++++
 cr-daily-branch             |    3 -
 make-bench-flight           |  153 +++++++++++++++++++++++++++++++
 mg-host-hw-specs            |   35 +++++++
 mg-kernbench-download       |   49 ++++++++++
 mg-unixbench-download       |   40 ++++++++
 sg-run-job                  |  102 ++++++++++++++++++++-
 standalone                  |    7 +
 standalone-reset            |    9 +-
 ts-bench-hostcmp-guest-prep |   87 ++++++++++++++++++
 ts-bench-hostcmp-host-prep  |   74 +++++++++++++++
 ts-bench-hostcmp-post       |   81 +++++++++++++++++
 ts-debian-fixup             |    3 +
 ts-debian-hvm-install       |    3 -
 ts-kernbench-build          |   86 ++++++++++++++++++
 ts-kernbench-reslts         |   75 +++++++++++++++
 ts-kernbench-run            |   63 +++++++++++++
 ts-unixbench-build          |   77 ++++++++++++++++
 ts-unixbench-reslts         |   84 +++++++++++++++++
 ts-unixbench-run            |   67 ++++++++++++++
 ts-xen-install              |   38 --------
 24 files changed, 1534 insertions(+), 68 deletions(-)
 create mode 100644 Osstest/Benchmarking.pm
 create mode 100755 make-bench-flight
 create mode 100755 mg-host-hw-specs
 create mode 100755 mg-kernbench-download
 create mode 100755 mg-unixbench-download
 create mode 100755 ts-bench-hostcmp-guest-prep
 create mode 100755 ts-bench-hostcmp-host-prep
 create mode 100755 ts-bench-hostcmp-post
 create mode 100755 ts-kernbench-build
 create mode 100755 ts-kernbench-reslts
 create mode 100755 ts-kernbench-run
 create mode 100755 ts-unixbench-build
 create mode 100755 ts-unixbench-reslts
 create mode 100755 ts-unixbench-run

<<This happens because I choose it to happen!>> (Raistlin Majere)
Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.