viridian time_ref_count triggers guest clock drift

[James Dingwall <james-xen@xxxxxxxxxxxxxx>]

Hi,

I've been looking at clock drift problem we've been having in Windows VMs
which seems to come down to whether the time_ref_count enlightenment is
enabled for the guest.  For migration compatibility reasons we had the
list expanded to:

viridian = ['base', 'freq', 'time_ref_count', 'apic_assist', 'crash_ctl']

The drift is no longer observed with:

viridian = ['base', 'freq', 'apic_assist', 'crash_ctl']


The drift is also absent with (enabled stime causes the guest gets stuck in
boot):

viridian = ['all', '!time_ref_count', '!stime']


The drift is observed with:

viridian = 1


The method of testing the drift was to execute the following command in
the guest:

w32tm /stripchart /computer:0.pool.ntp.org /rdtsc

first and last lines of output, default sample period is 2s, total 41 samples:

The current time is 15/05/2025 14:32:25.
RdtscStart, RdtscEnd, FileTime, RoundtripDelay, NtpOffset
391444616392, 391478090002, 133917895450376964, +00.0100979, +10.3926898
...
637570858836, 637609379234, 133917896256928566, +00.0135181, +17.8456872


Curiously the rate of drift is exacerbated by using 'spice = 1' approx
0.1s / s, vs 'spice = 0' approx 0.02s / s.  When 'time_ref_count' is set it
is possible to observe a higher than expected frequency in the guest 'System
Information' (also reported with get-wmiobject Win32_Processor -Property
CurrentClockSpeed) but the registry key 
HKEY_LOCAL_MACHINE\HARDWARE\DESCRIPTION\System\CentralProcessor\0
~MHz was set to the expected speed and agreed with `xl debug-key s` output.

Forcing the guest to use the platform clock (presumably using the hpet) also
prevented the clock from drifting.

bcdedit /set useplatformclock yes

Other options relating to "Guest Virtual Time Controls" were tested without
any option resolving the problem.  We could reproduce this with Intel and
AMD processors.

Guest operating system: Windows 2012, Windows 10

Xen version: 4.19.3-pre, 4.18.3, 4.15.4, 4.14.3 (our internal ticket was opened
for the two older releases but they haven't been checked again)

Xen params: console=vga,com2 console_timestamps=datems dom0_max_vcpus=4-8 
dom0_mem=min:6144,max:65536m iommu=on,required,intpost,verbose,debug x2apic=off 
sched=credit2 flask=enforcing gnttab_max_frames=128 xpti=off smt=on 
cpufreq=xen:performance spec-ctrl=gds-mit=0 com2=115200,8n1
Guest config:

memory = 2048
vcpus = 2
cpu_weight=256
pae = 1
acpi = 1
apic = 1
xen_platform_pci = 1
viridian = ['base', 'freq', 'apic_assist', 'crash_ctl']
vga = 'stdvga'
videoram = 16
soundhw = 'hda'
spice = 1
spicehost = '127.0.0.1'
spiceport = 35999
spicedisable_ticketing = 1
spicevdagent = 1
spice_clipboard_sharing = 0
spice_image_compression = 'auto_glz'
sdl = 0
vnc = 1
vncunused = 0
vnclisten = '0.0.0.0:99'
usb = 0
usbdevice = 'tablet'
keymap = 'en-us'
vif = [
    
'vifname=winguest{%domid}.0,script=vif-local,bridge=wan,mac=00:16:3e:01:6e:d8,backend=netdom'
]
name = 'winguest-00'
uuid = '08092537-70b6-4248-bfc4-4f6ecd92c230'
disk = [
    
'phy:/dev/zvol/ztank/08092537-70b6-4248-bfc4-4f6ecd92c230/c,hda,w,no-discard',
    
'phy:/dev/zvol/ztank/08092537-70b6-4248-bfc4-4f6ecd92c230/d,hdb,w,no-discard'
]
type = 'hvm'
dm_restrict = 1
device_model_chroot = 0
device_model_override = '/usr/lib/xen/bin/qemu-system-i386'
device_model_args = [
  '-object', 
'tls-creds-x509,id=tls0,endpoint=client,dir=/etc/certificates/usb,verify-peer=yes,sanity-check=no',
  # SERIAL
  '-chardev', 
'socket,id=charredir_serial0,host=127.0.0.1,port=48051,reconnect=2,nodelay=on,keepalive=on,user-timeout=3250',
  '-device', 'isa-serial,chardev=charredir_serial0',
  '-chardev', 
'socket,id=charredir_serial1,host=127.0.0.1,port=48052,reconnect=2,nodelay=on,keepalive=on,user-timeout=3250',
  '-device', 'isa-serial,chardev=charredir_serial1',
  '-chardev', 
'socket,id=charredir_serial2,host=127.0.0.1,port=48053,reconnect=2,nodelay=on,keepalive=on,user-timeout=3250',
  '-device', 'pci-serial,chardev=charredir_serial2',
  '-trace', 'events=/etc/xen/qemu-trace-options',
]
boot = 'cn'
localtime = 1
on_poweroff = 'destroy'
on_crash = 'preserve'
on_reboot = 'destroy'
seclabel = 'system_u:system_r:migrate_domU_t'


If there's any further information which would be useful please let me know.

Thanks,
James