[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] [xen-4.6-testing test] 107076: regressions - trouble: broken/fail/pass



flight 107076 xen-4.6-testing real [real]
http://logs.test-lab.xenproject.org/osstest/logs/107076/

Regressions :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
 test-amd64-i386-rumprun-i386 12 guest-destroy            fail REGR. vs. 106819
 test-xtf-amd64-amd64-3       10 xtf-fep                  fail REGR. vs. 106819
 test-xtf-amd64-amd64-3       11 xtf-run                  fail REGR. vs. 106819
 test-xtf-amd64-amd64-5       11 xtf-run                  fail REGR. vs. 106819
 test-xtf-amd64-amd64-1       10 xtf-fep                  fail REGR. vs. 106819
 test-xtf-amd64-amd64-2       11 xtf-run                  fail REGR. vs. 106819
 test-amd64-amd64-rumprun-amd64 12 guest-destroy          fail REGR. vs. 106819
 test-xtf-amd64-amd64-1       11 xtf-run                  fail REGR. vs. 106819
 test-xtf-amd64-amd64-4       10 xtf-fep                  fail REGR. vs. 106819
 test-amd64-amd64-xl-xsm      14 guest-saverestore        fail REGR. vs. 106819
 test-amd64-amd64-libvirt     14 guest-saverestore        fail REGR. vs. 106819
 test-amd64-i386-libvirt-xsm  14 guest-saverestore        fail REGR. vs. 106819
 test-amd64-amd64-migrupgrade 24 leak-check/check/dst_host fail REGR. vs. 106819
 test-xtf-amd64-amd64-4       11 xtf-run                  fail REGR. vs. 106819
 test-amd64-i386-qemuu-rhel6hvm-intel  9 redhat-install   fail REGR. vs. 106819
 test-amd64-i386-xl-xsm       14 guest-saverestore        fail REGR. vs. 106819
 test-amd64-amd64-xl          14 guest-saverestore        fail REGR. vs. 106819
 test-amd64-i386-xl           14 guest-saverestore        fail REGR. vs. 106819
 test-amd64-i386-xl-qemuu-ovmf-amd64 9 debian-hvm-install fail REGR. vs. 106819
 test-amd64-i386-freebsd10-i386 13 guest-saverestore      fail REGR. vs. 106819
 test-amd64-i386-qemuu-rhel6hvm-amd  9 redhat-install     fail REGR. vs. 106819
 test-amd64-amd64-xl-multivcpu 14 guest-saverestore       fail REGR. vs. 106819
 test-amd64-i386-freebsd10-amd64 13 guest-saverestore     fail REGR. vs. 106819
 test-amd64-amd64-xl-qemut-debianhvm-amd64 9 debian-hvm-install fail REGR. vs. 
106819
 test-amd64-amd64-qemuu-nested-intel 9 debian-hvm-install fail REGR. vs. 106819
 test-amd64-i386-libvirt      14 guest-saverestore        fail REGR. vs. 106819
 test-amd64-amd64-libvirt-pair 21 guest-migrate/src_host/dst_host fail REGR. 
vs. 106819
 test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm 9 debian-hvm-install fail 
REGR. vs. 106819
 test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm 9 debian-hvm-install 
fail REGR. vs. 106819
 test-amd64-amd64-xl-credit2  14 guest-saverestore        fail REGR. vs. 106819
 test-amd64-i386-migrupgrade 24 leak-check/check/dst_host fail REGR. vs. 106819
 test-amd64-i386-qemut-rhel6hvm-intel  9 redhat-install   fail REGR. vs. 106819
 test-amd64-amd64-pygrub       9 debian-di-install        fail REGR. vs. 106819
 test-amd64-amd64-amd64-pvgrub  9 debian-di-install       fail REGR. vs. 106819
 test-amd64-amd64-pair 21 guest-migrate/src_host/dst_host fail REGR. vs. 106819
 test-amd64-i386-qemut-rhel6hvm-amd  9 redhat-install     fail REGR. vs. 106819
 test-amd64-amd64-xl-qcow2     9 debian-di-install        fail REGR. vs. 106819
 test-amd64-i386-xl-qemut-debianhvm-amd64-xsm 9 debian-hvm-install fail REGR. 
vs. 106819
 test-amd64-amd64-xl-qemuu-debianhvm-amd64 9 debian-hvm-install fail REGR. vs. 
106819
 test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm 9 debian-hvm-install fail REGR. 
vs. 106819
 test-amd64-i386-pair  21 guest-migrate/src_host/dst_host fail REGR. vs. 106819
 test-amd64-amd64-libvirt-xsm 14 guest-saverestore        fail REGR. vs. 106819
 test-amd64-i386-xl-qemut-debianhvm-amd64 9 debian-hvm-install fail REGR. vs. 
106819
 test-amd64-i386-libvirt-pair 21 guest-migrate/src_host/dst_host fail REGR. vs. 
106819
 test-amd64-amd64-i386-pvgrub  9 debian-di-install        fail REGR. vs. 106819
 test-amd64-i386-xl-qemuu-debianhvm-amd64 9 debian-hvm-install fail REGR. vs. 
106819
 test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm 9 debian-hvm-install fail REGR. 
vs. 106819
 test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 9 debian-hvm-install fail 
REGR. vs. 106819
 test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm 9 debian-hvm-install fail 
REGR. vs. 106819
 test-amd64-amd64-xl-qemuu-ovmf-amd64 9 debian-hvm-install fail REGR. vs. 106819
 test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm 9 debian-hvm-install fail REGR. 
vs. 106819
 test-amd64-amd64-libvirt-vhd  9 debian-di-install        fail REGR. vs. 106819
 test-armhf-armhf-libvirt-xsm 14 guest-stop               fail REGR. vs. 106819
 test-amd64-i386-xl-raw        9 debian-di-install        fail REGR. vs. 106819
 test-armhf-armhf-xl-arndale 15 guest-start/debian.repeat fail REGR. vs. 106819
 test-armhf-armhf-libvirt     14 guest-stop               fail REGR. vs. 106819
 test-armhf-armhf-xl-credit2 15 guest-start/debian.repeat fail REGR. vs. 106819
 test-armhf-armhf-xl-xsm     15 guest-start/debian.repeat fail REGR. vs. 106819
 test-armhf-armhf-xl-cubietruck 15 guest-start/debian.repeat fail REGR. vs. 
106819
 test-armhf-armhf-xl-multivcpu 15 guest-start/debian.repeat fail REGR. vs. 
106819
 test-amd64-amd64-qemuu-nested-amd  9 debian-hvm-install  fail REGR. vs. 106819
 test-armhf-armhf-xl         15 guest-start/debian.repeat fail REGR. vs. 106819
 test-armhf-armhf-libvirt-raw  9 debian-di-install        fail REGR. vs. 106819
 test-armhf-armhf-xl-vhd       9 debian-di-install        fail REGR. vs. 106819
 test-amd64-amd64-xl-qemuu-winxpsp3  9 windows-install    fail REGR. vs. 106819
 test-amd64-amd64-xl-qemut-winxpsp3  9 windows-install    fail REGR. vs. 106819
 test-amd64-i386-xl-qemut-win7-amd64  9 windows-install   fail REGR. vs. 106819
 test-amd64-i386-xl-qemut-winxpsp3  9 windows-install     fail REGR. vs. 106819
 test-amd64-i386-xl-qemuu-winxpsp3-vcpus1 9 windows-install fail REGR. vs. 
106819
 test-amd64-i386-xl-qemuu-win7-amd64  9 windows-install   fail REGR. vs. 106819
 test-amd64-i386-xl-qemuu-winxpsp3  9 windows-install     fail REGR. vs. 106819
 test-amd64-i386-xl-qemut-winxpsp3-vcpus1 9 windows-install fail REGR. vs. 
106819
 test-amd64-amd64-xl-qemut-win7-amd64  9 windows-install  fail REGR. vs. 106819
 test-amd64-amd64-xl-qemuu-win7-amd64  9 windows-install  fail REGR. vs. 106819

Regressions which are regarded as allowable (not blocking):
 test-amd64-amd64-xl-rtds     14 guest-saverestore        fail REGR. vs. 106819
 test-armhf-armhf-libvirt-xsm 13 saverestore-support-check    fail  like 106819
 test-armhf-armhf-libvirt     13 saverestore-support-check    fail  like 106819
 test-armhf-armhf-xl-rtds     15 guest-start/debian.repeat    fail  like 106819

Tests which did not succeed, but are not blocking:
 test-amd64-amd64-libvirt     12 migrate-support-check        fail   never pass
 test-amd64-amd64-xl-pvh-intel 11 guest-start                  fail  never pass
 test-amd64-i386-libvirt-xsm  12 migrate-support-check        fail   never pass
 test-amd64-i386-libvirt      12 migrate-support-check        fail   never pass
 test-amd64-amd64-xl-pvh-amd  11 guest-start                  fail   never pass
 test-amd64-amd64-libvirt-xsm 12 migrate-support-check        fail   never pass
 test-armhf-armhf-libvirt-xsm 12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-arndale  12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-arndale  13 saverestore-support-check    fail   never pass
 test-armhf-armhf-xl-rtds     12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-rtds     13 saverestore-support-check    fail   never pass
 test-armhf-armhf-libvirt     12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-credit2  12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-credit2  13 saverestore-support-check    fail   never pass
 test-armhf-armhf-xl-cubietruck 12 migrate-support-check        fail never pass
 test-armhf-armhf-xl-cubietruck 13 saverestore-support-check    fail never pass
 test-armhf-armhf-xl-xsm      12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-xsm      13 saverestore-support-check    fail   never pass
 test-armhf-armhf-xl-multivcpu 12 migrate-support-check        fail  never pass
 test-armhf-armhf-xl-multivcpu 13 saverestore-support-check    fail  never pass
 test-armhf-armhf-xl          12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl          13 saverestore-support-check    fail   never pass

version targeted for testing:
 xen                  7ff6d9fc19ef88d2ca3304a312c0e8a46b61f546
baseline version:
 xen                  ac4c5d4ddf89051365da2acba5c6c306a10e0bbe

Last test of basis   106819  2017-03-21 19:13:35 Z   11 days
Testing same since   107020  2017-03-31 07:11:02 Z    1 days    3 attempts

------------------------------------------------------------
People who touched revisions under test:
  Daniel De Graaf <dgdegra@xxxxxxxxxxxxx>
  Dario Faggioli <dario.faggioli@xxxxxxxxxx>
  Jan Beulich <jbeulich@xxxxxxxx>
  Roger Pau Monné <roger.pau@xxxxxxxxxx>

jobs:
 build-amd64-xsm                                              pass    
 build-armhf-xsm                                              pass    
 build-i386-xsm                                               pass    
 build-amd64-xtf                                              pass    
 build-amd64                                                  pass    
 build-armhf                                                  pass    
 build-i386                                                   pass    
 build-amd64-libvirt                                          pass    
 build-armhf-libvirt                                          pass    
 build-i386-libvirt                                           pass    
 build-amd64-prev                                             pass    
 build-i386-prev                                              pass    
 build-amd64-pvops                                            pass    
 build-armhf-pvops                                            pass    
 build-i386-pvops                                             pass    
 build-amd64-rumprun                                          pass    
 build-i386-rumprun                                           pass    
 test-xtf-amd64-amd64-1                                       fail    
 test-xtf-amd64-amd64-2                                       fail    
 test-xtf-amd64-amd64-3                                       fail    
 test-xtf-amd64-amd64-4                                       fail    
 test-xtf-amd64-amd64-5                                       fail    
 test-amd64-amd64-xl                                          fail    
 test-armhf-armhf-xl                                          fail    
 test-amd64-i386-xl                                           fail    
 test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm                fail    
 test-amd64-i386-xl-qemut-debianhvm-amd64-xsm                 fail    
 test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm           fail    
 test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm            fail    
 test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm                fail    
 test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm                 fail    
 test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm        fail    
 test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm         fail    
 test-amd64-amd64-libvirt-xsm                                 fail    
 test-armhf-armhf-libvirt-xsm                                 fail    
 test-amd64-i386-libvirt-xsm                                  fail    
 test-amd64-amd64-xl-xsm                                      fail    
 test-armhf-armhf-xl-xsm                                      fail    
 test-amd64-i386-xl-xsm                                       fail    
 test-amd64-amd64-qemuu-nested-amd                            fail    
 test-amd64-amd64-xl-pvh-amd                                  fail    
 test-amd64-i386-qemut-rhel6hvm-amd                           fail    
 test-amd64-i386-qemuu-rhel6hvm-amd                           fail    
 test-amd64-amd64-xl-qemut-debianhvm-amd64                    fail    
 test-amd64-i386-xl-qemut-debianhvm-amd64                     fail    
 test-amd64-amd64-xl-qemuu-debianhvm-amd64                    fail    
 test-amd64-i386-xl-qemuu-debianhvm-amd64                     fail    
 test-amd64-i386-freebsd10-amd64                              fail    
 test-amd64-amd64-xl-qemuu-ovmf-amd64                         fail    
 test-amd64-i386-xl-qemuu-ovmf-amd64                          fail    
 test-amd64-amd64-rumprun-amd64                               fail    
 test-amd64-amd64-xl-qemut-win7-amd64                         fail    
 test-amd64-i386-xl-qemut-win7-amd64                          fail    
 test-amd64-amd64-xl-qemuu-win7-amd64                         fail    
 test-amd64-i386-xl-qemuu-win7-amd64                          fail    
 test-armhf-armhf-xl-arndale                                  fail    
 test-amd64-amd64-xl-credit2                                  fail    
 test-armhf-armhf-xl-credit2                                  fail    
 test-armhf-armhf-xl-cubietruck                               fail    
 test-amd64-i386-freebsd10-i386                               fail    
 test-amd64-i386-rumprun-i386                                 fail    
 test-amd64-amd64-qemuu-nested-intel                          fail    
 test-amd64-amd64-xl-pvh-intel                                fail    
 test-amd64-i386-qemut-rhel6hvm-intel                         fail    
 test-amd64-i386-qemuu-rhel6hvm-intel                         fail    
 test-amd64-amd64-libvirt                                     fail    
 test-armhf-armhf-libvirt                                     fail    
 test-amd64-i386-libvirt                                      fail    
 test-amd64-amd64-migrupgrade                                 broken  
 test-amd64-i386-migrupgrade                                  broken  
 test-amd64-amd64-xl-multivcpu                                fail    
 test-armhf-armhf-xl-multivcpu                                fail    
 test-amd64-amd64-pair                                        fail    
 test-amd64-i386-pair                                         fail    
 test-amd64-amd64-libvirt-pair                                fail    
 test-amd64-i386-libvirt-pair                                 fail    
 test-amd64-amd64-amd64-pvgrub                                fail    
 test-amd64-amd64-i386-pvgrub                                 fail    
 test-amd64-amd64-pygrub                                      fail    
 test-amd64-amd64-xl-qcow2                                    fail    
 test-armhf-armhf-libvirt-raw                                 fail    
 test-amd64-i386-xl-raw                                       fail    
 test-amd64-amd64-xl-rtds                                     fail    
 test-armhf-armhf-xl-rtds                                     fail    
 test-amd64-i386-xl-qemut-winxpsp3-vcpus1                     fail    
 test-amd64-i386-xl-qemuu-winxpsp3-vcpus1                     fail    
 test-amd64-amd64-libvirt-vhd                                 fail    
 test-armhf-armhf-xl-vhd                                      fail    
 test-amd64-amd64-xl-qemut-winxpsp3                           fail    
 test-amd64-i386-xl-qemut-winxpsp3                            fail    
 test-amd64-amd64-xl-qemuu-winxpsp3                           fail    
 test-amd64-i386-xl-qemuu-winxpsp3                            fail    


------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images

Logs, config files, etc. are available at
    http://logs.test-lab.xenproject.org/osstest/logs

Explanation of these reports, and of osstest in general, is at
    http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
    http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master

Test harness code can be found at
    http://xenbits.xen.org/gitweb?p=osstest.git;a=summary


Not pushing.

------------------------------------------------------------
commit 7ff6d9fc19ef88d2ca3304a312c0e8a46b61f546
Author: Dario Faggioli <dario.faggioli@xxxxxxxxxx>
Date:   Fri Mar 31 09:03:32 2017 +0200

    xen: sched: don't call hooks of the wrong scheduler via VCPU2OP
    
    Within context_saved(), we call the context_saved hook,
    and we use VCPU2OP() to determine from what scheduler.
    VCPU2OP uses DOM2OP, which uses d->cpupool, which is
    NULL when d is the idle domain. And in that case,
    DOM2OP just returns ops, the scheduler of cpupool0.
    
    Therefore, if:
    - cpupool0's scheduler defines context_saved (like
      Credit2 and RTDS do),
    - we are not in cpupool0 (i.e., our scheduler is
      not ops),
    - we are context switching from idle,
    
    we call VCPU2OP(idle_vcpu), which means
    DOM2OP(idle->cpupool), which is ops.
    
    Therefore, we both:
    - check if context_saved is defined in the wrong
      scheduler;
    - if yes, call the wrong one.
    
    When using Credit2 at boot, and also Credit2 in
    the other cpupool, this is wrong but innocuous,
    because it only involves the idle vcpus.
    
    When using Credit2 at boot, and Credit1 in the
    other cpupool, this is *totally* wrong, and
    it's by chance it does not explode!
    
    When using Credit2 and other schedulers I'm
    developping, I hit the following assert (in
    sched_credit2.c, on a CPU inside a cpupool that
    does not use Credit2):
    
    csched2_context_saved()
    {
     ...
     ASSERT(!vcpu_on_runq(svc));
     ...
    }
    
    Fix this by dealing explicitly, in VCPU2OP, with
    idle vcpus, returning the scheduler of the pCPU
    they (always) run on.
    
    Signed-off-by: Dario Faggioli <dario.faggioli@xxxxxxxxxx>
    Reviewed-by: Juergen Gross <jgross@xxxxxxxx>
    Reviewed-by: George Dunlap <george.dunlap@xxxxxxxxxx>
    master commit: a3653e6a279213ba4e883b2252415dc98633106a
    master date: 2017-03-27 14:28:05 +0100

commit 7017321d9b7b23adb50cad66d55ae526ab5d9fe1
Author: Jan Beulich <jbeulich@xxxxxxxx>
Date:   Fri Mar 31 09:03:01 2017 +0200

    x86/EFI: avoid Xen image when looking for module/kexec position
    
    When booting straight from EFI, we don't further try to relocate Xen.
    As a result, so far we also didn't avoid the area Xen uses when looking
    for a location to put modules or the kexec area. Introduce a fake
    module slot to deal with that without having to fiddle with a lot of
    code.
    
    Signed-off-by: Jan Beulich <jbeulich@xxxxxxxx>
    Reviewed-by: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
    master commit: e22e1c47958a4778cd7baa3980f74e52f525ba28
    master date: 2017-03-20 09:27:12 +0100

commit 541ad61a921965c692ea31a06d9d14a1bb1e7346
Author: Jan Beulich <jbeulich@xxxxxxxx>
Date:   Fri Mar 31 09:01:29 2017 +0200

    x86/EFI: avoid overrunning mb_modules[]
    
    Commit 436fb462ab ("x86/microcode: enable boot time (pre-Dom0)
    loading") added a 4th module without providing an array slot for it.
    
    Signed-off-by: Jan Beulich <jbeulich@xxxxxxxx>
    Reviewed-by: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
    master commit: 02b37b7eff39e40828041b2fe480725ab8443258
    master date: 2017-03-17 15:45:22 +0100

commit 4f9617120b09ad1554d8e4a0ca817e27bfbbe1a1
Author: Roger Pau Monné <roger.pau@xxxxxxxxxx>
Date:   Fri Mar 31 09:00:50 2017 +0200

    build/clang: fix XSM dummy policy when using clang 4.0
    
    There seems to be some weird bug in clang 4.0 that prevents xsm_pmu_op from
    working as expected, and vpmu.o ends up with a reference to
    __xsm_action_mismatch_detected which makes the build fail:
    
    [...]
    ld    -melf_x86_64_fbsd  -T xen.lds -N prelink.o  \
        xen/common/symbols-dummy.o -o xen/.xen-syms.0
    prelink.o: In function `xsm_default_action':
    xen/include/xsm/dummy.h:80: undefined reference to 
`__xsm_action_mismatch_detected'
    xen/xen/include/xsm/dummy.h:80: relocation truncated to fit: R_X86_64_PC32 
against undefined symbol `__xsm_action_mismatch_detected'
    ld: xen/xen/.xen-syms.0: hidden symbol `__xsm_action_mismatch_detected' 
isn't defined
    
    Then doing a search in the objects files:
    
    # find xen/ -type f -name '*.o' -print0 | xargs -0 bash -c \
      'for filename; do nm "$filename" | \
      grep -q __xsm_action_mismatch_detected && echo "$filename"; done' bash
    xen/arch/x86/prelink.o
    xen/arch/x86/cpu/vpmu.o
    xen/arch/x86/cpu/built_in.o
    xen/arch/x86/built_in.o
    
    The current patch is the only way I've found to fix this so far, by simply
    moving the XSM_PRIV check into the default case in xsm_pmu_op. This also 
fixes
    the behavior of do_xenpmu_op, which will now return -EINVAL for unknown
    XENPMU_* operations, instead of -EPERM when called by a privileged domain.
    
    Signed-off-by: Roger Pau Monné <roger.pau@xxxxxxxxxx>
    Acked-by: Daniel De Graaf <dgdegra@xxxxxxxxxxxxx>
    master commit: 9e4d116faff4545a7f21c2b01008e94d68e6db58
    master date: 2017-03-14 18:19:29 +0100

commit 9eb0aa2f47a78ffbab9e7ab9974440a132ce073f
Author: Roger Pau Monné <roger.pau@xxxxxxxxxx>
Date:   Fri Mar 31 09:00:13 2017 +0200

    x86: drop unneeded __packed attributes
    
    There where a couple of unneeded packed attributes in several x86-specific
    structures, that are obviously aligned. The only non-trivial one is
    vmcb_struct, which has been checked to have the same layout with and without
    the packed attribute using pahole. In that case add a build-time size check 
to
    be on the safe side.
    
    No functional change is expected as a result of this commit.
    
    Signed-off-by: Roger Pau Monné <roger.pau@xxxxxxxxxx>
    Reviewed-by: Jan Beulich <jbeulich@xxxxxxxx>
    Reviewed-by: Boris Ostrovsky <boris.ostrovsky@xxxxxxxxxx>
    master commit: 4036e7c592905c2292cdeba8269e969959427237
    master date: 2017-03-07 17:11:06 +0100
(qemu changes not included)

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.