[Xen-devel] Reducing impact of save/restore/dump on Dom0

["Graham, Simon" <Simon.Graham@xxxxxxxxxxx>]

Currently, save, restore and dump all used cached I/O in Dom0 to
write/read the file containing the memory image of the DomU - when the
memory assigned to the DomU is greater than free memory in Dom0, this
leads to severe memory thrashing and generally the Dom0 performance goes
into the toilet.

The 'classic' answer to avoiding this when writing very large files is,
of course, to use non-cached I/O to manipulate the files - this
introduces the restriction that reads and writes have to be at sector
aligned offsets and be an integral number of sectors in length. I've
been working on a prototype for using O_DIRECT for the dump case; this
one is actually easier to do because the on-disk format of a dump file
already has the memory pages aligned to sector boundaries so only the
dump header has to be handled specially - attached is a patch to
unstable for this code which makes the following changes to
tools/libxc/xc_core.c:

1. xc_domain_dumpcore_via_callback is modified to ensure that the buffer
containing
   page data to be dumped is page aligned (using mmap instead of malloc)
2. xc_domain_dumpcore is modified to open the file O_DIRECT and to
buffer writes issued
   by xc_domain_dumpcore_via_callback as needed.

I'd welcome comments on this prior to submitting it for inclusion. I
know that there is a lot of discussion around O_DIRECT being a bad idea
in general and that the POSIX fadvise() call should provide the hints to
tell Linux that a file's cache doesn't need to be maintained but from
looking at the 2.6.16 & 2.6.18 sources I believe this call is not fully
implemented. I also think that this is a case where you never want to
use the file cache at all.

Solving this problem for save/restore is more tricky because the
on-disk/wire save format does not force the page data to be page aligned
-- my proposal would be to page align each batch of pages written,
leaving pad between the batch header and the pages themselves but I
realize that a change in on-disk/wire format is a potential
compatibility problem.

Simon

Attachment: direct-io-dump.patch
Description: direct-io-dump.patch

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel