[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] Clock jumping during live migrations


  • To: xen-devel@xxxxxxxxxxxxxxxxxxx
  • From: Nathan March <nathan@xxxxxx>
  • Date: Wed, 26 Oct 2011 15:31:38 -0700
  • Delivery-date: Wed, 26 Oct 2011 15:32:34 -0700
  • Domainkey-signature: a=rsa-sha1; c=nofws; d=gt.net; h=message-id:date :from:mime-version:to:subject:content-type :content-transfer-encoding; q=dns; s=mail; b=osmjkkbXOpxs5Af6CNC itW13Ct/a9JLGAJ6vrr79mdHNSC82+S7gaVU8lbwp08KqHtgFGeUM8jjfznd8P6i 93Nro2oqfZ6sW5rK5JENlQC/s43pTupOVKpRKOROBrP+2gt87j9JrA6UVWmRN5TV k8mbi8gZySK1S4dVhloQjhCI=
  • List-id: Xen developer discussion <xen-devel.lists.xensource.com>

Hi everyone,

Having an issue where the clock on a VM will jump ahead by the timezone offset but only certain directions between certain machines.

For example, if I run a "sleep 1; date" loop on the VM and migrate it from xen4 to xen7:

Wed Oct 26 21:14:46 PDT 2011
Thu Oct 27 04:19:03 PDT 2011

Trying multiple times with the same dom0's it seems to be consistently 7 hours (PST offset maybe?) 4 minutes and 17 seconds:

Thu Oct 27 04:23:27 PDT 2011
Thu Oct 27 11:27:45 PDT 2011

Thu Oct 27 11:43:01 PDT 2011
Thu Oct 27 18:47:18 PDT 2011

If I keep migrating back and forth it keeps going ahead each time, it never snaps back. Those were both migrating from xen4 -> xen7, migrating the VM back to xen4 does not result in a clock change of any noticeable amount.

I've tried using hwclock to make sure the RTC's are in sync with the system clock but it didn't seem to make a difference (although they were wrong initially, unless xen requires a reboot to pick up a change here).

If I try from xen8 to xen7 it does it, but by a much smaller amount... again, going the reverse direction causes no change.

Thu Oct 27 18:50:08 PDT 2011
Thu Oct 27 18:51:57 PDT 2011

and again the time between multiple runs seems to be a consistent 1:50:

Fri Oct 28 16:16:27 PDT 2011
Fri Oct 28 16:18:17 PDT 2011

All the dom0's are identical as far as I'm aware except that some have hyperthreading on and some have it off. I've seen this issue between two hyperthreading enabled hosts so doesn't seem to be specific to moving between them.

I've tested this with the VM using both the xen and the tsc clocksource but it made no difference.

Other info for xen4 / xen7:

xen4 ~ # uname -a
Linux xen4 3.0.3 #3 SMP Thu Sep 1 23:39:43 PDT 2011 x86_64 Intel(R) Xeon(R) CPU L5640 @ 2.27GHz GenuineIntel GNU/Linux
xen4 ~ # uptime
 14:48:33 up 1 day,  6:03,  2 users,  load average: 0.00, 0.01, 0.05
xen4 ~ # xm info
host                   : xen4
release                : 3.0.3
version                : #3 SMP Thu Sep 1 23:39:43 PDT 2011
machine                : x86_64
nr_cpus                : 12
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 1
cpu_mhz                : 2266
hw_caps : bfebfbff:2c100800:00000000:00003f40:009ee3fd:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 98294
free_memory            : 87822
free_cpus              : 0
xen_major              : 4
xen_minor              : 1
xen_extra              : .1
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline : console=com1,com2,vga com1=115200,8n1 com2=115200,8n1 dom0_mem=1024M dom0_max_vcpus=1 dom0_vcpus_pin=true
cc_compiler            : gcc version 4.3.4 (Gentoo 4.3.4 p1.1, pie-10.1.5)
cc_compile_by          : root
cc_compile_domain      : nmsrv.com
cc_compile_date        : Thu Sep  1 23:34:06 PDT 2011
xend_config_format     : 4
xen4 ~ # cat /etc/timezone
PST8PDT
xen4 ~ # md5sum /etc/localtime
c9452f6b9e08d83c6815c38600798964  /etc/localtime

xen7 ~ # uname -a
Linux xen7 3.0.3 #2 SMP Tue Aug 23 12:58:39 PDT 2011 x86_64 Intel(R) Xeon(R) CPU L5640 @ 2.27GHz GenuineIntel GNU/Linux
xen7 ~ # uptime
 14:48:34 up 42 days, 23:38,  3 users,  load average: 0.04, 0.08, 0.13
xen7 ~ # xm info
host                   : xen7
release                : 3.0.3
version                : #2 SMP Tue Aug 23 12:58:39 PDT 2011
machine                : x86_64
nr_cpus                : 24
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 2
cpu_mhz                : 2266
hw_caps : bfebfbff:2c100800:00000000:00003f40:009ee3fd:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 98294
free_memory            : 24329
free_cpus              : 0
xen_major              : 4
xen_minor              : 1
xen_extra              : .1
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline : console=com1,com2,vga com1=115200,8n1 com2=115200,8n1 dom0_mem=1024M dom0_max_vcpus=1 dom0_vcpus_pin=true
cc_compiler            : gcc version 4.3.4 (Gentoo 4.3.4 p1.1, pie-10.1.5)
cc_compile_by          : root
cc_compile_domain      : nmsrv.com
cc_compile_date        : Mon Aug 22 11:28:50 PDT 2011
xend_config_format     : 4
xen7 ~ # cat /etc/timezone
PST8PDT
xen7 ~ # md5sum /etc/localtime
c9452f6b9e08d83c6815c38600798964  /etc/localtime

and the test VM i've been using:

nathanxen1 ~ # uname -a
Linux nathanxen1 2.6.32.27-grsec-gt-r2 #1 SMP Wed Aug 10 17:13:56 PDT 2011 x86_64 Intel(R) Xeon(R) CPU L5640 @ 2.27GHz GenuineIntel GNU/Linux nathanxen1 ~ # cat /sys/devices/system/clocksource/clocksource0/available_clocksource
xen tsc
nathanxen1 ~ # cat /sys/devices/system/clocksource/clocksource0/current_clocksource
tsc

Any suggestions?

Unfortunately I don't know when this bug appeared, it used to work fine so not sure what I upgraded to give me an idea of when it was introduced.

Thanks,
Nathan

--
Nathan March<nathan@xxxxxx>
Gossamer Threads Inc. http://www.gossamer-threads.com/
Tel: (604) 687-5804 Fax: (604) 687-5806


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.