Bug involving %fs register and suspend-to-RAM

["M. Vefa Bicakci" <m.v.b@xxxxxxxxxx>]

Hello all,

I encountered an unusual bug involving Xen's and Linux's handling of the
%fs register after resuming from suspend-to-RAM (S2R) with Xen 4.8.y,
and I am reaching out to the xen-devel mailing list, because the bug
appears to affect Xen's master branch too.

In summary, Xen uses/overwrites the %fs register during the resume from
S2R in "xen/arch/x86/boot/wakeup.S", and this register's value is not
restored from the saved context during the context switch to dom0 after
the resume.  Furthermore, Linux running in dom0 appears to leak the %fs
register's value to systemd-sleep, which in turn encounters a
segmentation fault when it attempts to access a variable in thread local
storage. The fault is intermittent. A segfault is observed approximately
once in 10~15 suspend and resume cycles.

Here is the problematic code in context from Xen 4.8.y, where the %fs
register is used:

  xen/arch/x86/boot/wakeup.S
  37
  38  1:      # Show some progress if VGA is resumed
  39          movw    $0xb800, %ax
  40          movw    %ax, %fs
  41          movw    $0x0e00 + 'L', %fs:(0x10)
  42
  43          # boot trampoline is under 1M, and shift its start into
  44          # %fs to reference symbols in that area
  45          mov     wakesym(trampoline_seg), %fs
  46          lidt    %fs:bootsym(idt_48)
  47          lgdt    %fs:bootsym(gdt_48)

And here are some logs obtained from an instrumented version of Linux.
(The magic value 0x9b00 originates from line 45 in the code excerpt
above, which was verified by instrumenting Xen.)

  kernel: Enabling non-boot CPUs ...
  kernel: installing Xen timer for CPU 1
  kernel: xxx: save_fsgs: cpu:0 task:systemd-sleep saved fsindex != 0 (0x9b00)
  kernel: xxx: load_seg_legacy: cpu:0 prev_task:systemd-sleep next_task:swapper/0 
next_index: 0 <= 3
  kernel: xxx: load_seg_legacy: cpu:0 prev_task:systemd-sleep 
next_task:swapper/0 next_base == 0. next_index:0x0 CPU has X86_BUG_NULL_SEG
  kernel: xxx: load_seg_legacy: loading segments. cpu:0. prev_task:swapper/0: 
next_task:systemd-sleep May cause a segfault... next_index: 39680 > 3 (0x9b00)
  ...
  kernel: PM: suspend exit
  kernel: systemd-sleep[7102]: segfault at 10 ip 00007fb79ac283f9 sp 
00007ffd7c1dd508 error 4 in libc-2.24.so[7fb79abb1000+1bd000]

I encountered this bug while testing patches that I had cherry-picked to
Qubes OS's Xen 4.8.y tree to reduce power consumption after S2R by
parking unused CPU hyperthreads. Interestingly, if the changes to park
the CPU's hyperthreads are reverted, then the bug cannot be reproduced
(at least in my experience).

Note that this issue appears to affect recent Xen versions as well. The
%fs register continues to be used in "wakeup.S" to report progress on
the console/VGA, as can be seen in the code listing above. With Xen
4.8.y, unconditionally restoring %fs in the function
restore_rest_processor_state (in file xen/arch/x86/acpi/suspend.c) is
sufficient to resolve this issue.

Some of my debugging notes can be found at [1], and a patch that
resolves this issue in Xen 4.8.y (along with cherry-picked patches to
park hyperthreads) can be found at [2].

All this to say, there appears to be a bug that might negatively affect
Xen's master branch as well, and I wanted to bring this to the attention
of the xen-devel mailing list's subscribers, on the recommendation of
Marek Marczykowski-Górecki of the Qubes OS team.

If there is any further information or assistance I can provide, I would
be happy to help. I would also appreciate any feedback regarding the
last patch at [2] (i.e.,
"patch-0009-x86-acpi-suspend-Unconditionally-restore-fs.patch").

Thank you,

Vefa

[1] https://github.com/QubesOS/qubes-issues/issues/5210
[2] 
https://github.com/m-v-b/qubes-vmm-xen/commit/9678563a9d099db2879af9c534dbc655ea086d17