[Xen-devel] Ehancement to domU suspend/resume

["Tian, Kevin" <kevin.tian@xxxxxxxxx>]

Hi, all,
        When working on adding PM support to xen, we realized that 
some enhancements are required to suspend/resume domU. Following 
is some background and thoughts, and welcome on comments. :-)

        Currently we use a simple approach (pause/unpause) for domU 
when ready to pull whole platform into a power save state, saying a 
S3. Because pause/unpause is out of domU's knowledge, domU 
observes soft lockup when unpaused after resuming from S3. Also 
this can not handle driver domain case. We tried "xm save/restore" 
or equivalent "xm suspend/resume", it works however overhead is a 
bit high since the whole domU memory is saved to disk and domain 
itself is destroyed after suspend. For S3 support, it's better to have 
quick suspend/resume, cause memory still keeps along the process.

        Basically the change may lie with two aspects:
                - Lightweight "xm suspend/resume"
                - Extend suspend support to driver domU

[Lightweight "xm suspend/resume"]
        It's reasonable for current implementation to save and release 
whole memory of domU after suspended, since it allows more 
memory available to other domains. However for platform level S3, 
this is redundant when box is physically put into a suspend state. 
What we need is just to send a suspend notification into domU, and 
let domU fall into __xen_suspend path. Then domU exits scheduler 
by issuing HYPERVISOR_suspend. Nothing else required after this.
 After resume, domU just continues to run after suspend point.

        Even __xen_suspend path is a bit heavy, and in this case 
resources don't change for domU even after resume. Maybe we 
can benefit from recent checkpoint patch which has appropriate 
change on this path.

        But I'm not familiar with control panel side and hope some guys 
can suggest me. My gut-feeling is to add an option (like -L) to existing

"xm suspend/resume", instead of a new command. Actually the 
possible changes may look like what checkpoint patch does except 
no immediate resume and we need disable memory save logic.

[Driver domU suspend]
        This has to be added if one device is assigned to a domU and 
we want system still working correctly after resume. When driver 
domU receives suspend request, it should invoke driver suspend 
method of owned physical devices. Before that, one other necessary 
step is to freeze all processes since some may still hold critical 
resource. In this case, we need borrow some Linux PM stuff into 
xen suspend path, something like:
        Smp_suspend();
        freeze_processes();
        device_suspend();
        device_power_down();
        xenbus_suspend();
        ...
        HYPERVISOR_suspend(virt_to_mfn(xen_start_info));
        ...
        xenbus_resume();
        device_power_up();
        device_resume();
        thaw_processes();
        smp_resume();

        It may be more difficult if we want to support wake-on-LAN when 
that NIC is assigned to domU, which is more tightly related to ACPI. 
So we will simply consider normal device suspend here.

        One interesting question is, why doesn't current __xen_suspend 
freeze running processes? My rough feeling is that virtual device state 
is still kept (like in xenstore, BE, etc.) along with single domU 
suspend/resume, and thus almost all front-end drivers (except TPM) 
have no suspend method. If no driver suspend methods are invoked, 
no need to freeze processes since no contention there. Correct me for 
the real trick. :-)

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel