[Xen-devel] [Question] How to support page offline in Xen environment

["Jiang, Yunhong" <yunhong.jiang@xxxxxxxxx>]

Hi, all
        Page offline can be used by many purpose, like memory offline, memory 
power management, proactive action when multiple CE error happen to one page 
etc.  In virtualization environment without guest offline support, we think 
offline a page usually means replace the old page with a new one transparently 
to guest. 
        Currently we are trying to add page offline support in Xen environment 
. We'd share our idea with the mailing list before we begin implment it and 
hope to get feedback from the community.

Our idea:
        Page offline will be done in two steps: firstly, a page is marked 
offline-pending (if the page is free already, it will be marked as offline 
directly); secondly, when that page is freed, that page will be marked offline 
automatically, and will not allocated anymore. Notice is, we will not support 
all types of page because the diffrent page usage model. Basically, xen heap 
page, page owned by dom0 can not be offline. Also,  it is complex to offline a 
page that is used for guest with device assigned.

        We are considering utilize the live migration mechanism to achieve the 
two steps. The user space tools will firstly mark page offline_pending through 
hypercall, this hyercall will also return the owners of the pages. secondly, if 
all pages can be offlined, user space tools should live migrate the domains 
owning those pages. Thirdly, user space tools will check all page is offlined 
already.

        Following hypercall will be added:
        int xen_page_offline_pending(int start_pfn,  int end_pfn, void *result, 
void *owners)
        IN: start_pfn/end_pfn: 
                the range of pages to be offlined.
        OUT: result: 
                A buffer contain the page status for each page, it can be:
                        offlined: the page is offlined already (e.g. the page 
is already freed when the hypercall happen)
                        offline_pending: the page will be offline when freed
                        offline_fail: The page can't be offline, may because it 
is used by xen/dom0. Notice is, if any page is marked offline_fail, this 
hypercall will not change any page's status (i.e. no page will be marked 
offline_pending or offlined) to make sure atomic operation.
                        other status: Other status to be defined in future.
        OUT: owners:
                A buffer contains the domains owning of the pages. Because of 
security consideration, it will not state which domain owning which page.

        Need notice is, issue exists for the live migrate mechanism:
        a) the domain ID will be changed after live migrate
        b) live migration will fail for a domain with device assigned, so user 
space tools have to hot remove the device, or fail the page offline requirement.


Some other option:
        Of course, there are still exists some other mechianism to achive this 
purpose:
        1) Handle the page offline requirement in Xen environment. It is simple 
to page offline a HVM domain ( without any device assigned) utilize the p2m 
table, and re-create the shadow/EPT table from scratch. But  it is not so easy 
for PV domain (maybe we can switch the domain to shadow mode during this 
procedure), and domain with device assigned. Also, it is complex to support 
following page types:
        a) page shared between multiple domain, like granted to dom0,
        b) page used for domain control, for example, the page used for hvm 
domain's vlapic.
        
Any feedback is welcome.

Thanks
Yunhong Jiang
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel