[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-devel] Re: [PATCH v3] [linux-3.0+ for xen] tmem: self-ballooning and frontswap-selfshrinking
On Wed, Jun 29, 2011 at 01:15:24PM -0700, Dan Magenheimer wrote: > Hi Konrad (and other reviewers) -- Sorry for late reply, however, I am very busy now. > Here is version 3 of the Xen tmem selfballooning/frontswap-selfshrinking > patch. Many thanks to the reviewers of V1 and V2 for your excellent > comments and feedback. I hope V3 is ready for merging as I'm > hoping to get this merged for the upcoming 3.1 window. > > If it is satisfactory, you may wish to check the commit in my tree: > > git://git.kernel.org/pub/scm/linux/kernel/git/djm/tmem.git > #xen-tmem-selfballoon-v3 > > If all looks good and there are no issues raised in the next > day or two, I will post an official git pull request to you > and cc lkml. > > Note that this patch has a slight build dependency on last week's: > > afec6e04922d0c8c7e244be2e544bac5e7e36294 xen: prepare tmem shim to handle > frontswap > > in your devel/next-3.0 branch (because tmem_enabled is EXPORT_SYM'ed > in that commit and extern'ed in a file in this commit). I don't > think this is a problem as you've already pushed that commit > to linux-next, but am noting the dependency in case some kind > of commit reordering were to occur. > > === > >From: Dan Magenheimer <dan.magenheimer@xxxxxxxxxx> > Subject: xen: tmem: self-ballooning and frontswap-selfshrinking > > This patch introduces two in-kernel drivers for Xen transcendent memory > ("tmem") functionality that complement cleancache and frontswap. Both > use control theory to dynamically adjust and optimize memory utilization. > Selfballooning controls the in-kernel Xen balloon driver, targeting a goal > value (vm_committed_as), thus pushing less frequently used clean > page cache pages (through the cleancache code) into Xen tmem where > Xen can balance needs across all VMs residing on the physical machine. > Frontswap-selfshrinking controls the number of pages in frontswap, > driving it towards zero (effectively doing a partial swapoff) when > in-kernel memory pressure subsides, freeing up RAM for other VMs. > > More detail is provided in the header comment of xen-selfballooning.c. > > Signed-off-by: Dan Magenheimer <dan.magenheimer@xxxxxxxxxx> > > [v3: konrad.wilk@xxxxxxxxxx: fix potential divides-by-zero] > [v3: konrad.wilk@xxxxxxxxxx: add many more comments, fix nits] > [v2: rebased to linux-3.0-rc1] > [v2: Ian.Campbell@xxxxxxxxxx: reorganize as new file (xen-selfballoon.c)] > [v2: dkiper@xxxxxxxxxxxx: proper access to vm_committed_as] > [v2: dkiper@xxxxxxxxxxxx: accounting fixes] > Cc: Jan Beulich <JBeulich@xxxxxxxxxx> > Cc: Jeremy Fitzhardinge <jeremy@xxxxxxxx> > Cc: <xen-devel@xxxxxxxxxxxxxxxxxxx> > > --- > > Diffstat: > drivers/xen/Kconfig | 12 > drivers/xen/Makefile | 1 > drivers/xen/xen-balloon.c | 2 > drivers/xen/xen-selfballoon.c | 429 +++++++++++++++++++++ > include/xen/balloon.h | 10 > include/xen/tmem.h | 5 > 6 files changed, 459 insertions(+) > > diff -Napur -X linux-3.0-rc1/Documentation/dontdiff > linux-3.0-rc1-frontswap/drivers/xen/Kconfig > linux-3.0-rc1-frontswap-selfballoon/drivers/xen/Kconfig > --- linux-3.0-rc1-frontswap/drivers/xen/Kconfig 2011-05-31 > 17:09:07.606910896 -0600 > +++ linux-3.0-rc1-frontswap-selfballoon/drivers/xen/Kconfig 2011-06-29 > 10:52:39.834899532 -0600 > @@ -9,6 +9,18 @@ config XEN_BALLOON > the system to expand the domain's memory allocation, or alternatively > return unneeded memory to the system. > > +config XEN_SELFBALLOONING > + bool "dynamically self-balloon kernel memory to target" > + depends on XEN && XEN_BALLOON && CLEANCACHE > + default n > + help > + Self-ballooning dynamically balloons available kernel memory driven > + by the current usage of anonymous memory ("committed AS") and > + controlled by various sysfs-settable parameters. May be overridden > + by the noselfballooning kernel boot parameter. If FRONTSWAP > + is configured, also enables frontswap-selfshrinking, which can be > + overridden by the noselfshrink kernel boot paramater. > + > config XEN_SCRUB_PAGES > bool "Scrub pages before returning them to system" > depends on XEN_BALLOON > diff -Napur -X linux-3.0-rc1/Documentation/dontdiff > linux-3.0-rc1-frontswap/drivers/xen/Makefile > linux-3.0-rc1-frontswap-selfballoon/drivers/xen/Makefile > --- linux-3.0-rc1-frontswap/drivers/xen/Makefile 2011-05-31 > 17:09:57.006875306 -0600 > +++ linux-3.0-rc1-frontswap-selfballoon/drivers/xen/Makefile 2011-06-16 > 11:39:06.123852289 -0600 > @@ -8,6 +8,7 @@ obj-$(CONFIG_BLOCK) += biomerge.o > obj-$(CONFIG_HOTPLUG_CPU) += cpu_hotplug.o > obj-$(CONFIG_XEN_XENCOMM) += xencomm.o > obj-$(CONFIG_XEN_BALLOON) += xen-balloon.o > +obj-$(CONFIG_XEN_SELFBALLOONING) += xen-selfballoon.o > obj-$(CONFIG_XEN_DEV_EVTCHN) += xen-evtchn.o > obj-$(CONFIG_XEN_GNTDEV) += xen-gntdev.o > obj-$(CONFIG_XEN_GRANT_DEV_ALLOC) += xen-gntalloc.o > diff -Napur -X linux-3.0-rc1/Documentation/dontdiff > linux-3.0-rc1-frontswap/drivers/xen/xen-balloon.c > linux-3.0-rc1-frontswap-selfballoon/drivers/xen/xen-balloon.c > --- linux-3.0-rc1-frontswap/drivers/xen/xen-balloon.c 2011-05-29 > 18:43:36.000000000 -0600 > +++ linux-3.0-rc1-frontswap-selfballoon/drivers/xen/xen-balloon.c > 2011-06-20 15:37:30.405798859 -0600 > @@ -98,6 +98,8 @@ static int __init balloon_init(void) > > register_balloon(&balloon_sysdev); > > + register_xen_selfballooning(&balloon_sysdev); > + > target_watch.callback = watch_target; > xenstore_notifier.notifier_call = balloon_init_watcher; > > diff -Napur -X linux-3.0-rc1/Documentation/dontdiff > linux-3.0-rc1-frontswap/drivers/xen/xen-selfballoon.c > linux-3.0-rc1-frontswap-selfballoon/drivers/xen/xen-selfballoon.c > --- linux-3.0-rc1-frontswap/drivers/xen/xen-selfballoon.c 1969-12-31 > 17:00:00.000000000 -0700 > +++ linux-3.0-rc1-frontswap-selfballoon/drivers/xen/xen-selfballoon.c > 2011-06-29 11:44:44.067872414 -0600 > @@ -0,0 +1,429 @@ > +/****************************************************************************** > + * Xen selfballoon driver (and optional frontswap self-shrinking driver) > + * > + * Copyright (c) 2009-2011, Dan Magenheimer, Oracle Corp. > + * > + * This code complements the cleancache and frontswap patchsets to optimize > + * support for Xen Transcendent Memory ("tmem"). The policy it implements > + * is rudimentary and will likely improve over time, but it does work well > + * enough today. > + * > + * Two functionalities are implemented here which both use "control theory" > + * (feedback) to optimize memory utilization. In a virtualized environment > + * such as Xen, RAM is often a scarce resource and we would like to ensure > + * that each of a possibly large number of virtual machines is using RAM > + * efficiently, i.e. using as little as possible when under light load > + * and obtaining as much as possible when memory demands are high. > + * Since RAM needs vary highly dynamically and sometimes dramatically, > + * "hysteresis" is used, that is, memory target is determined not just > + * on current data but also on past data stored in the system. > + * > + * "Selfballooning" creates memory pressure by managing the Xen balloon > + * driver to decrease and increase available kernel memory, driven > + * largely by the target value of "Committed_AS" (see /proc/meminfo). > + * Since Committed_AS does not account for clean mapped pages (i.e. pages > + * in RAM that are identical to pages on disk), selfballooning has the > + * affect of pushing less frequently used clean pagecache pages out of > + * kernel RAM and, presumably using cleancache, into Xen tmem where > + * Xen can more efficiently optimize RAM utilization for such pages. > + * > + * When kernel memory demand unexpectedly increases faster than Xen, via > + * the selfballoon driver, is able to (or chooses to) provide usable RAM, > + * the kernel may invoke swapping. In most cases, frontswap is able > + * to absorb this swapping into Xen tmem. However, due to the fact > + * that the kernel swap subsystem assumes swapping occurs to a disk, > + * swapped pages may sit on the disk for a very long time; even if > + * the kernel knows the page will never be used again. This is because > + * the disk space costs very little and can be overwritten when > + * necessary. When such stale pages are in frontswap, however, they > + * are taking up valuable real estate. "Frontswap selfshrinking" works > + * to resolve this: When frontswap activity is otherwise stable > + * and the guest kernel is not under memory pressure, the frontswap > + * selfshrinking" accounts for this by providing pressure to remove some > + * pages from frontswap and return them to kernel memory. > + * > + * For both "selfballooning" and "frontswap-selfshrinking", worker > + * threads are used and sysfs tunables are provided to adjust the frequency > + * and rate of adjustments to achieve the goal, as well as to disable one > + * or both functions independently. > + * > + * While some argue that this functionality can and should be implemented > + * in userspace, it has been observed that bad things happen (e.g. OOMs). > + * > + */ > + > +#include <linux/kernel.h> > +#include <linux/mm.h> > +#include <linux/mman.h> > + > +#include <xen/balloon.h> > + > +#ifdef CONFIG_FRONTSWAP > +#include <linux/frontswap.h> > +#endif Please move this condition to linux/frontswap.h and include this header file (linux/frontswap.h) unconditionally. > +#include <xen/tmem.h> > + > +/* enable/disable with sysfs */ > +static int xen_selfballooning_enabled __read_mostly; > + > +/* enable/disable with kernel boot option */ > +static bool __initdata use_selfballooning = true; static bool use_selfballooning __initdata = true; Please look into include/linux/init.h for details. > + > +/* > + * controls rate at which memory target (this iteration) approaches > + * ultimate goal when memory need is increasing (up-hysteresis) or > + * decreasing (down-hysteresis). higher values of hysteresis cause > + * slower increases/decreases > + */ > +static unsigned int selfballoon_downhysteresis __read_mostly; > +static unsigned int selfballoon_uphysteresis __read_mostly; > + > +/* in HZ, controls frequency of worker invocation */ > +static unsigned int selfballoon_interval __read_mostly; Could you create a struct selfballoon ??? I think it will be more readable. > + > +static void selfballoon_process(struct work_struct *work); > +static DECLARE_DELAYED_WORK(selfballoon_worker, selfballoon_process); > + > +#ifdef CONFIG_FRONTSWAP > +/* enable/disable with sysfs */ > +static bool frontswap_selfshrinking __read_mostly; > + > +/* enable/disable with kernel boot option */ > +static bool __initdata use_frontswap_selfshrink = true; Ditto. > + > +/* control rate for frontswap shrinking. higher hysteresis is slower */ > +static unsigned int frontswap_hysteresis __read_mostly; > + > +/* > + * number of selfballoon worker invocations to wait before observing that > + * frontswap selfshrinking should commence. (Note that selfshrinking does > + * not use a separate worker thread.) > + */ > +static unsigned int frontswap_inertia __read_mostly; > + > +/* countdown to next invocation of frontswap_shrink() */ > +static unsigned long frontswap_inertia_counter; struct frontswap ??? > + > +/* > + * Invoked by the selfballoon worker thread, uses current number of pages > + * in frontswap (frontswap_curr_pages()), previous status, and control > + * values (hysteresis and inertia) to determine if frontswap should be > + * shrunk and what the new frontswap size should be. Note that > + * frontswap_shrink is essentially a partial swapoff that immediately > + * transfers pages from the "swap device" (frontswap) back into kernel > + * RAM; despite the name, frontswap "shrinking" is very different from > + * the "shrinker" interface used by the kernel MM subsystem to reclaim > + * memory. > + */ > +static void frontswap_selfshrink(void) > +{ > + static unsigned long cur_frontswap_pages; > + static unsigned long last_frontswap_pages; > + static unsigned long tgt_frontswap_pages; If you create struct frontswap then move it to this struct. > + > + if (!frontswap_selfshrinking) > + return; > + last_frontswap_pages = cur_frontswap_pages; > + cur_frontswap_pages = frontswap_curr_pages(); > + if (!cur_frontswap_pages || > + (cur_frontswap_pages > last_frontswap_pages)) { > + frontswap_inertia_counter = frontswap_inertia; > + return; > + } > + if (frontswap_inertia_counter && --frontswap_inertia_counter) > + return; > + if (cur_frontswap_pages <= frontswap_hysteresis) > + tgt_frontswap_pages = 0; > + else > + tgt_frontswap_pages = cur_frontswap_pages - > + (cur_frontswap_pages / frontswap_hysteresis); > + frontswap_shrink(tgt_frontswap_pages); > +} > + > +static int __init no_frontswap_selfshrink_setup(char *s) > +{ > + use_frontswap_selfshrink = false; > + return 1; > +} > + > +__setup("noselfshrink", no_frontswap_selfshrink_setup); > +#endif > + > +/* > + * use current balloon size, the goal (vm_committed_as), and hysteresis > + * parameters to set a new target balloon size > + */ > +static void selfballoon_process(struct work_struct *work) > +{ > + unsigned long cur_pages, goal_pages, tgt_pages; > + bool reset_timer = false; > + > + if (xen_selfballooning_enabled) { > + cur_pages = balloon_stats.current_pages; > + tgt_pages = cur_pages; /* default is no change */ > + goal_pages = percpu_counter_read_positive(&vm_committed_as) + > + balloon_stats.current_pages - totalram_pages; > + if (cur_pages > goal_pages) > + tgt_pages = cur_pages - > + ((cur_pages - goal_pages) / > + selfballoon_downhysteresis); > + else if (cur_pages < goal_pages) > + tgt_pages = cur_pages + > + ((goal_pages - cur_pages) / > + selfballoon_uphysteresis); > + /* else if cur_pages == goal_pages, no change */ > + balloon_set_new_target(tgt_pages); > + reset_timer = true; > + } > +#ifdef CONFIG_FRONTSWAP > + if (frontswap_selfshrinking) { > + frontswap_selfshrink(); > + reset_timer = true; > + } > +#endif > + if (reset_timer) > + schedule_delayed_work(&selfballoon_worker, > + selfballoon_interval * HZ); > +} > + > +#ifdef CONFIG_SYSFS > + > +#include <linux/sysdev.h> > +#include <linux/capability.h> > + > +#define SELFBALLOON_SHOW(name, format, args...) > \ > + static ssize_t show_##name(struct sys_device *dev, \ > + struct sysdev_attribute *attr, \ > + char *buf) \ > + { \ > + return sprintf(buf, format, ##args); \ > + } > + > +SELFBALLOON_SHOW(selfballooning, "%d\n", xen_selfballooning_enabled); > + > +static ssize_t store_selfballooning(struct sys_device *dev, > + struct sysdev_attribute *attr, > + const char *buf, > + size_t count) > +{ > + char *endchar; > + bool was_enabled = xen_selfballooning_enabled; > + > + if (!capable(CAP_SYS_ADMIN)) > + return -EPERM; > + > + xen_selfballooning_enabled = !!memparse(buf, &endchar); Replace memparse() by strict_strtoul(). > + > + if (!was_enabled && xen_selfballooning_enabled) > + schedule_delayed_work(&selfballoon_worker, > + selfballoon_interval * HZ); > + > + return count; > +} > + > +static SYSDEV_ATTR(selfballooning, S_IRUGO | S_IWUSR, > + show_selfballooning, store_selfballooning); > + > +SELFBALLOON_SHOW(selfballoon_interval, "%d\n", selfballoon_interval); > + > +static ssize_t store_selfballoon_interval(struct sys_device *dev, > + struct sysdev_attribute *attr, > + const char *buf, > + size_t count) > +{ > + char *endchar; > + unsigned int val; > + > + if (!capable(CAP_SYS_ADMIN)) > + return -EPERM; > + val = memparse(buf, &endchar); Ditto. > + if (val != 0) > + selfballoon_interval = val; > + return count; > +} > + > +static SYSDEV_ATTR(selfballoon_interval, S_IRUGO | S_IWUSR, > + show_selfballoon_interval, store_selfballoon_interval); > + > +SELFBALLOON_SHOW(selfballoon_downhys, "%d\n", selfballoon_downhysteresis); > + > +static ssize_t store_selfballoon_downhys(struct sys_device *dev, > + struct sysdev_attribute *attr, > + const char *buf, > + size_t count) > +{ > + char *endchar; > + unsigned int val; > + > + if (!capable(CAP_SYS_ADMIN)) > + return -EPERM; > + val = memparse(buf, &endchar); Ditto. > + if (val != 0) > + selfballoon_downhysteresis = val; > + return count; > +} > + > +static SYSDEV_ATTR(selfballoon_downhysteresis, S_IRUGO | S_IWUSR, > + show_selfballoon_downhys, store_selfballoon_downhys); > + > + > +SELFBALLOON_SHOW(selfballoon_uphys, "%d\n", selfballoon_uphysteresis); > + > +static ssize_t store_selfballoon_uphys(struct sys_device *dev, > + struct sysdev_attribute *attr, > + const char *buf, > + size_t count) > +{ > + char *endchar; > + unsigned int val; > + > + if (!capable(CAP_SYS_ADMIN)) > + return -EPERM; > + val = memparse(buf, &endchar); Ditto. > + if (val != 0) > + selfballoon_uphysteresis = val; > + return count; > +} > + > +static SYSDEV_ATTR(selfballoon_uphysteresis, S_IRUGO | S_IWUSR, > + show_selfballoon_uphys, store_selfballoon_uphys); > + > +#ifdef CONFIG_FRONTSWAP > +SELFBALLOON_SHOW(frontswap_selfshrinking, "%d\n", frontswap_selfshrinking); > + > +static ssize_t store_frontswap_selfshrinking(struct sys_device *dev, > + struct sysdev_attribute *attr, > + const char *buf, > + size_t count) > +{ > + char *endchar; > + bool was_enabled = frontswap_selfshrinking; > + > + if (!capable(CAP_SYS_ADMIN)) > + return -EPERM; > + > + frontswap_selfshrinking = !!memparse(buf, &endchar); Ditto. > + > + if (!was_enabled && !xen_selfballooning_enabled && > + frontswap_selfshrinking) > + schedule_delayed_work(&selfballoon_worker, > + selfballoon_interval * HZ); I think it is not worth to wrap lines which only have langht slightly above 80 characters limit. In this case two lines are more readable. > + > + return count; > +} > + > +static SYSDEV_ATTR(frontswap_selfshrinking, S_IRUGO | S_IWUSR, > + show_frontswap_selfshrinking, store_frontswap_selfshrinking); > + > +SELFBALLOON_SHOW(frontswap_inertia, "%d\n", frontswap_inertia); > + > +static ssize_t store_frontswap_inertia(struct sys_device *dev, > + struct sysdev_attribute *attr, > + const char *buf, > + size_t count) > +{ > + char *endchar; > + unsigned long val; > + > + if (!capable(CAP_SYS_ADMIN)) > + return -EPERM; > + val = memparse(buf, &endchar); Ditto. > + if (val != 0) { > + frontswap_inertia = val; > + frontswap_inertia_counter = val; > + } > + return count; > +} > + > +static SYSDEV_ATTR(frontswap_inertia, S_IRUGO | S_IWUSR, > + show_frontswap_inertia, store_frontswap_inertia); > + > +SELFBALLOON_SHOW(frontswap_hysteresis, "%d\n", frontswap_hysteresis); > + > +static ssize_t store_frontswap_hysteresis(struct sys_device *dev, > + struct sysdev_attribute *attr, > + const char *buf, > + size_t count) > +{ > + char *endchar; > + unsigned int val; > + > + if (!capable(CAP_SYS_ADMIN)) > + return -EPERM; > + val = memparse(buf, &endchar); Ditto. > + if (val != 0) > + frontswap_hysteresis = val; > + return count; > +} > + > +static SYSDEV_ATTR(frontswap_hysteresis, S_IRUGO | S_IWUSR, > + show_frontswap_hysteresis, store_frontswap_hysteresis); > + > +#endif /* CONFIG_FRONTSWAP */ > + > +static struct attribute *selfballoon_attrs[] = { > + &attr_selfballooning.attr, > + &attr_selfballoon_interval.attr, > + &attr_selfballoon_downhysteresis.attr, > + &attr_selfballoon_uphysteresis.attr, > +#ifdef CONFIG_FRONTSWAP > + &attr_frontswap_selfshrinking.attr, > + &attr_frontswap_hysteresis.attr, > + &attr_frontswap_inertia.attr, > +#endif > +}; > + > +static struct attribute_group selfballoon_group = { > + .name = "selfballoon", > + .attrs = selfballoon_attrs > +}; > +#endif > + > +int register_xen_selfballooning(struct sys_device *sysdev) > +{ > + int error = -1; int error = -ENOSYS; > + > +#ifdef CONFIG_SYSFS > + error = sysfs_create_group(&sysdev->kobj, &selfballoon_group); > +#endif > + return error; > +} > +EXPORT_SYMBOL(register_xen_selfballooning); > + > +static int __init noselfballooning_setup(char *s) > +{ > + use_selfballooning = false; > + return 1; > +} > + > +__setup("noselfballooning", noselfballooning_setup); > + > +/* > + * the default values for the various parameters were deemed reasonable > + * by experimentation, may be workload-dependent, and can all be > + * adjusted via sysfs > + */ > +static int __init xen_selfballoon_init(void) > +{ > + if (!xen_domain()) > + return -ENODEV; > + > + pr_info("xen/balloon: Initializing Xen selfballooning driver.\n"); > + xen_selfballooning_enabled = tmem_enabled && use_selfballooning; > + selfballoon_interval = 5; > + selfballoon_downhysteresis = 8; > + selfballoon_uphysteresis = 1; > +#ifdef CONFIG_FRONTSWAP > + pr_info("xen/balloon: Initializing frontswap selfshrinking driver.\n"); > + frontswap_selfshrinking = use_frontswap_selfshrink && frontswap_enabled; > + frontswap_hysteresis = 20; > + frontswap_inertia = 3; > +#endif > + schedule_delayed_work(&selfballoon_worker, selfballoon_interval * HZ); > + > + return 0; > +} > + > +subsys_initcall(xen_selfballoon_init); > + > +MODULE_LICENSE("GPL"); > diff -Napur -X linux-3.0-rc1/Documentation/dontdiff > linux-3.0-rc1-frontswap/include/xen/balloon.h > linux-3.0-rc1-frontswap-selfballoon/include/xen/balloon.h > --- linux-3.0-rc1-frontswap/include/xen/balloon.h 2011-05-29 > 18:43:36.000000000 -0600 > +++ linux-3.0-rc1-frontswap-selfballoon/include/xen/balloon.h 2011-06-20 > 14:58:24.975176230 -0600 > @@ -23,3 +23,13 @@ void balloon_set_new_target(unsigned lon > > int alloc_xenballooned_pages(int nr_pages, struct page** pages); > void free_xenballooned_pages(int nr_pages, struct page** pages); > + > +struct sys_device; > +#ifdef CONFIG_XEN_SELFBALLOONING > +extern int register_xen_selfballooning(struct sys_device *sysdev); > +#else > +static inline int register_xen_selfballooning(struct sys_device *sysdev) > +{ > + return -1; return -ENOSYS; or return 0; > +} > +#endif > diff -Napur -X linux-3.0-rc1/Documentation/dontdiff > linux-3.0-rc1-frontswap/include/xen/tmem.h > linux-3.0-rc1-frontswap-selfballoon/include/xen/tmem.h > --- linux-3.0-rc1-frontswap/include/xen/tmem.h 1969-12-31 > 17:00:00.000000000 -0700 > +++ linux-3.0-rc1-frontswap-selfballoon/include/xen/tmem.h 2011-06-29 > 10:30:11.113929299 -0600 > @@ -0,0 +1,5 @@ > +#ifndef _XEN_TMEM_H > +#define _XEN_TMEM_H > +/* defined in drivers/xen/tmem.c */ > +extern int tmem_enabled; > +#endif /* _XEN_TMEM_H */ Daniel _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |