Re: [Xen-devel] Ever increasing time offset for HVM domain / Huge amounts of drift

[Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>]

> From: Phil Evans [mailto:Phil.Evans@xxxxxxxx]
> Sent: Monday, January 14, 2013 9:53 AM
> To: Pasi Kärkkäinen
> Cc: xen-devel@xxxxxxxxxxxxx
> Subject: Re: [Xen-devel] Ever increasing time offset for HVM domain / Huge 
> amounts of drift
> 
> I had tried messing with all but tsc_mode.  I have tried all options for 
> tsc_mode as well now but
> there is no change.  The key thing here is that the drift is increasing at a 
> rate of 1 second per
> second of uptime amounting to huge amounts.  It doesn't seem to be a clock 
> skew issue, it seems to be
> something is simply not counting at all.

Just a wild idea... What Windows version are you running?  Historically,
I think, Windows has ignored TSC i.e. never issues a rdtsc nor writes
to TSC.  I wonder if the kernel of whatever version of Windows you are
running (or the NTP sync somehow with the blessing of the kernel) _is_
checking certain hardware settings, then writing to the guest's (virtual)
TSC and this might cause things to get very confused.

P.S. Avoiding top-posting can be difficult and annoying if
you are running some versions of Outlook as your mail client.
Google it to see how, if you'd like to avoid complaints
on this list.

> From: Pasi Kärkkäinen [pasik@xxxxxx]
> Sent: 14 January 2013 16:30
> To: Phil Evans
> Cc: xen-devel@xxxxxxxxxxxxx
> Subject: Re: [Xen-devel] Ever increasing time offset for HVM domain / Huge 
> amounts of drift
> 
> On Mon, Jan 14, 2013 at 04:17:39PM +0000, Phil Evans wrote:
> > Hi,
> >
> > Sorry I should have included that in the first place:
> >
> 
> Ok. Did you try experimenting with these options?:
> 
> timer_mode=X
> hpet=0|1
> tsc_mode=X
> 
> 
> -- Pasi
> 
> 
> > import os,re
> > arch = os.uname()[4]
> > kernel = '/usr/lib/xen-default/boot/hvmloader'
> > builder = 'hvm'
> > name = 'vm_141'
> > memory = '2048'
> > disk = 
> > ['phy:/dev/storage_node_2/disk_806,xvda,w','file:/control/isos/empty.iso,xvdd:cdrom,r']
> > vif = ['mac=00:16:3e:3f:2f:1a, bridge=vlan_369, vifname=vm_141.0, 
> > ip=89.238.190.22 2a02:40:501:3::2
> 2a02:40:501:3::5 89.238.190.88','mac=00:16:3e:67:66:71, bridge=vlan_4000, 
> vifname=vm_141.1,
> ip=0.0.0.0','mac=00:16:3e:01:7e:e8, bridge=vlan_369, vifname=vm_141.2']
> > device_model = '/usr/lib/xen-default/bin/qemu-dm'
> > boot = 'cd'
> > vnc = 1
> > vncpasswd = 'YpD5aVZ8'
> > usbdevice = 'tablet'
> > acpi = 0
> > vcpus = 4
> > viridian = 1
> >
> > Thanks,
> > Phil.
> > ________________________________________
> > From: Pasi Kärkkäinen [pasik@xxxxxx]
> > Sent: 14 January 2013 15:14
> > To: Phil Evans
> > Cc: xen-devel@xxxxxxxxxxxxx
> > Subject: Re: [Xen-devel] Ever increasing time offset for HVM domain / Huge 
> > amounts of drift
> >
> > On Mon, Jan 14, 2013 at 01:37:01PM +0000, Phil Evans wrote:
> > > Hi,
> > >
> > > I am currently running Xen 4.2.1 (this has also happened in 4.2.0 as 
> > > well).  We have been having a
> major problem with sometimes huge amounts of clock drift in Windows VMs.  
> Sometimes the clock on a VM
> could suddenly jump by over a week (usually forwards, however time has been 
> known to go backwards as
> well).
> > >
> > > Now I don?t profess to know the internals of Xen, however through my 
> > > investigation I believe I
> have a degree of knowledge of what could be causing the problem.
> > >
> > > The steps to reproduce this (for me at least), is to simply do a manual 
> > > NTP sync on a Windows VM.
> Upon monitoring the qemu-dm log file for the VM, I see similar to the 
> following:
> > >
> > > Time offset set 489, added offset 480
> > > Time offset set 436, added offset -53
> > > Time offset set 496, added offset 60
> > > Time offset set 494, added offset -2
> > > Time offset set 554, added offset 60
> > > Time offset set 565, added offset 11
> > > Time offset set 606, added offset 41
> > > Time offset set -1974, added offset -2580
> > > Time offset set 1626, added offset 3600
> > > Time offset set 1579, added offset -47
> > > Time offset set 1639, added offset 60
> > >
> > > It seems to add the same number of seconds to the offset as has passed 
> > > since the last sync.  The
> offset just keeps on increasing, eventually resulting in huge numbers 
> equating to days.  Occasionally
> the offset may jump a bit and go down but the general trend is up.  Although 
> this does not affect the
> VM immediately, at some point I am guessing it syncs itself with the CMOS 
> clock (which is now a large
> number of seconds offset from the actual time), resulting in a huge jump in 
> time.  A reboot is a
> guaranteed way to get the new, incorrect time.
> > >
> > > Although I do not understand all of the underlying code, I presume the 
> > > correct way this should
> work is it should be comparing the CMOS time that?s just been set with the 
> hardware clock on the
> physical machine, resulting in an offset between the two.  This would result 
> in a generally stable
> number (ideally 0).  Obviously it is incorrect behaviour for the number to 
> keep going up.  To my mind
> it looks like it may be somehow getting an inaccurate time from the system 
> (in many cases a fixed time
> rather than an up-to-date current time).
> > >
> > > Does anyone have any light they may be able to shed on this?  Is it 
> > > possible it could be
> struggling to get an accurate time from the hardware?  I have checked on 
> several occasions and both
> the system time and the BIOS clock are spot on.
> > >
> >
> > Please paste the cfgfile of your HVM Windows.
> >
> > -- Pasi
> >
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel