Re: [Xen-devel] Xen balloon driver improvement (version 1)

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 22/10/14 17:29, Wei Liu wrote:
> Hi all
>
> This is my initial design to improve Xen balloon driver.
>
> PDF version with graphs can be found at
>
> http://xenbits.xen.org/people/liuw/xen-balloon-driver-improvement.pdf

+1 pandoc  (makes far nicer pdfs)

I have a few style recommendations, and some grammar issues

>
> % Xen Balloon Driver Improvement
> % Wei Liu <<wei.liu2@xxxxxxxxxx>>

% Draft $N

Helps here.

>
> -------------------------------------------
> Version     Date         Changes
> -------     ----         ------------------
>   1         22/10/2014   Initial version.
> -------------------------------------------
>
> ## Motives
>
> 1. Balloon pages fragments guest physical address space.

[style] For the benefit of people reading the text rather than the pdf,
properly numbering lists like these is nice.

> 1. Balloon compaction infrastructure can migrate ballooned pages from
>    start of zone to end of zone, hence creating contiguous guest physical
>    address space.

What does start of zone and end of zone mean?  While this document is
labelled "Xen balloon driver improvements", it applies mainly to the
Linux balloon driver, and should be noted as such.

> 1. Having contiguous guest physical address enables some options to
>    improve performance.
>
> ## Goal of improvement
>
> Balloon driver makes use of as many huge pages as possible,

"The balloon driver"

> defragmenting both guest address space and Xen pages. This should be

Defragmentation (i.e. actively undoing fragmentation) is different to
"preventing fragmentation".

In this case, the balloon driver defragmenting the guest physical
address space permits superpage ballooning which helps prevent host
fragmentation.

I think you should also note that both guest and host address
fragmentation cause performance issues, and this design covers
specifically guest fragmentation.

> achieved without any particular hypervisor side feature.
>
> ## Design and implementation
>
> When balloon driver is asked to increase / decrease reservation, it

"When the balloon driver"

> will always start with huge page. However, due to resource

"always start with a huge page"

> availability in both hypervisor and guest, it's not always possible to
> get hold of a huge page. In that case the driver will fall back to use
> normal size page. Balloon driver later will try to coalesce small size
> pages into huge page. As time goes by, both Xen and guest should use
> more and more huge pages.
>
> To achieve the said goal, several changes will be made:
>
> 1. Make use of balloon page compaction.
> 1. Maintain multiple queues for pages of different sizes and purposes.
> 1. Periodically exchange normal size pages with huge pages.
>
> ### Make use of balloon page compaction
>
> Balloon page migration moves balloon pages from start of zone to end
> of zone, making guest physical address space contiguous. This gives
> worker thread to allocate huge pages in order to coalesce small pages.

I can't parse this final sentence.

>
> Currently, Xen balloon driver gets its page directly from page
> allocator. To enable balloon page migration, those pages now need to
> be allocated from core balloon driver. Pages allocated from core
> balloon driver are subject to balloon page compaction.
>
> The use of balloon compaction doesn't require introducign new
> interfaces between Xen balloon driver and the rest of the system. Most
> changes are internal to Xen balloon driver.

The Linux driver.  The principle applies to each OS, but interaction
with the core memory allocator is very certainly system specific.

>
> Xen balloon driver will also need to provide a callback to migrate
> balloon page. In essence callback function receives "old page", which
> is a already ballooned out page, and "new page", which is a page to be
> ballooned out, then it inflates "old page" and deflates "new page".
>
> The core of migration callback is XENMEM\_exchange hypercall. This
> makes sure that inflation of old page and deflation of new page is
> done atomically, so even if a domain is beyond its memory target and
> being enforced, it can still compact memory.

"and the target is being enforced"

>
> ### Maintain multiple queues for pages of different sizes and purposes
>
> We maintain multiple queues for pages of different sizes inside Xen
> balloon driver, so that Xen balloon worker thread can coalesce smaller
> size pages into one larger size page. Queues for special purposed
> pages, such as balloon pages used to map foreign pages, are also
> maintained. These special purposed pages are not subject to migration
> and page coalescence.
>
> For instance, balloon driver can maintain three queues:
>
> 1. queue for 2 MB pages
> 1. queue for 4 KB pages (delegated to core balloon driver)
> 1. queue for pages used to mapped pages from other domain

What about 1GB pages?

>
> More queues can be added when necessary, but for now one queue for
> normal pages and one queue for huge page should be enough.
>
> ### Worker thread to coalesce small size pages
>
> Worker thread wakes up periodically to check if there's enough pages

"there are enough pages"

> in normal size page queue to coalesce into a huge page. If so, it will
> try to exchange that huge page into a number of normal size pages with
> XENMEM\_exchange hypercall.

Your diagram says "exchange 2M and 4K pages".  Exchange how, because you
cannot exchange a set of scattered 4K pages for a 2M contiguous one in
an HVM guest.

~Andrew


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel