(auto-)ballooning issue

[Jan Beulich <jbeulich@xxxxxxxx>]

Hello,

the other day I observed a strange issue with XTF's selftest failing to
have its VMs created, when running all flavors in a group. (Other tests
look to be similarly affected, it's just the selftest that I run most
frequently.) Originally I suspected a PVH-specific issue, but the
problem surfacing only there is because with PVH Dom0 I have less free
memory left after boot than with PV Dom0. Beyond that, both
configurations use the same hypervisor, with built-in DOM0_MEM="-255M".

The issue looks to be further affected (but not caused) by domain
cleanup being quite a bit slower under PVH Dom0, compared to PV. I.e.
by the time the 2nd test is started, memory from the 1st one still
wasn't completely freed. The result is that randomly one of the latter
(batched) tests fails at domain creation ("failed to free memory for
the domain").

xl's freemem() calls libxl_set_memory_target() followed by
libxl_wait_for_memory_target(). The latter function expects the domain
to balloon down enough for its ->tot_pages (in the hypervisor) to be
at or below the previously set target. However, already immediately
after boot "xl list -l" and "xs ls /" show target values which are 1
page below the hypervisor's record. With libxl_set_memory_target()
requesting relative adjustment, the Dom0 kernel will balloon out the
requested number of pages, but ->tot_pages going down by as many pages
isn't enough to please libxl_wait_for_memory_target().

I'm not even close to having an opinion as to where the problem is: It
could be that the kernel's balloon driver is off by a page. I'm more
inclined though to think that it is entirely unrealistic to expect the
kernel's balloon driver and Xen to have an exactly matching view of
the memory owned by the domain. Yet then it is simply invalid to
compare values taken from Xenstore against values taken from Xen. While
problematic for absolute requests, for relative ones it should be
possible apply the decrement to the source later used to compare
against while waiting.

Jan