Re: [Xen-ia64-devel] [PATCH] Use saner dom0 memory and vcpu defaults, don't panic on over-allocation

[Jarod Wilson <jwilson@xxxxxxxxxx>]

Alex Williamson wrote:
> On Mon, 2007-08-06 at 10:11 -0400, Jarod Wilson wrote:
>> Alex Williamson wrote: 
>>>    I think we're going to have to go with something like this, but why
>>> would we reduce the cap to 64MB?  I usually think of ia64 systems as
>>> having "bigger" I/O than x86, so it seems like maybe we want to stick
>>> with at least 128MB(?)
>> I think my original thought was that since your 96G box only needed
>> really 17MB, 64MB was a ton to withhold. But after hitting send, I was
>> thinking that no cap at all might make more sense -- you'd need 288GB of
>> RAM to even get to 64MB here, and that's a tiny drop in the bucket when
>> you have that much. Even with 1TB of RAM, we would still withhold less
>> than 256MB. Until we have some 1TB+ systems to test on, we don't really
>> know if reserving more than 128MB makes sense or not... I'd have to lean
>> toward simply not capping this withholding at all, at least for right now.
> 
>    That sounds ok with me, we can continue to fine tune it via bug
> reports if it's insufficient.

Okay, attaching one more rendition of the patch that goes this route.

> [snip]
>> But then I don't know what the typical end-user use case of ia64 xen is,
>> or what the average system size is, which would seem to be a fairly
>> relevant factor in deciding the most reasonable defaults... My intuition
>> says that most of our customers would like to have their systems boot up
>> using all available resources though. That, and I'm already starting to
>> feel like a 4G/4cpu default is a bit wimpy. :)
> 
>   It's ok to assign all memory and cpu to dom0 if you intend to use dom0
> as a compute server, but that's not really the way dom0 is meant to be
> used in a virtualized environment.  IIRC, dom0 has some scheduling
> preference over domUs, and can therefore starve or reduce the
> performance of the virtualized guests.  It's recommended that dom0 is
> primarily used as a management domain for the other guests and not used
> as your primary workload server.  IMHO, reducing the default set of
> resources available to dom0 encourages this behavior.  Thanks,

Also leaving the default a 4G/4cpu. I understand the argument for
keeping the allocations minimal and how dom0 *should* be used, I just
anticipate customers wondering where all their resources went. We can
just relnote it though, I think this is a reasonable compromise.

-- 
Jarod Wilson
jwilson@xxxxxxxxxx

Some ia64 xen dom0 tweaks:
* Increase default memory allocation from 512M to 4G
* Increase default vcpu allocation from 1 to 4
* Implement rough calculation of what the maximum memory
  that can be safely allocated to dom0 is
* If need be, scale down requested memory allocation to fit
  available memory, rather than simply panicking
* If dom0_mem=0 is specified, allocate all available mem

Signed-off-by: Jarod Wilson <jwilson@xxxxxxxxxx>

diff -r 039f2ccb1e38 xen/arch/ia64/xen/domain.c
--- a/xen/arch/ia64/xen/domain.c        Tue Jul 31 10:30:40 2007 -0600
+++ b/xen/arch/ia64/xen/domain.c        Tue Aug 07 16:41:39 2007 -0400
@@ -52,10 +52,11 @@
 #include <asm/perfmon.h>
 #include <public/vcpu.h>
 
-static unsigned long __initdata dom0_size = 512*1024*1024;
+/* dom0_size: default memory allocation for dom0 (~4GB) */
+static unsigned long __initdata dom0_size = 4096UL*1024UL*1024UL;
 
 /* dom0_max_vcpus: maximum number of VCPUs to create for dom0.  */
-static unsigned int __initdata dom0_max_vcpus = 1;
+static unsigned int __initdata dom0_max_vcpus = 4;
 integer_param("dom0_max_vcpus", dom0_max_vcpus); 
 
 extern char dom0_command_line[];
@@ -1195,8 +1196,41 @@ static void __init loaddomainelfimage(st
        }
 }
 
-void __init alloc_dom0(void)
-{
+static void __init calc_dom0_size(void)
+{
+       unsigned long domheap_pages;
+       unsigned long p2m_pages;
+       unsigned long spare_hv_pages;
+       unsigned long max_dom0_size;
+
+       /* Estimate maximum memory we can safely allocate for dom0
+        * by subtracting the p2m table allocation and a chunk of memory
+        * for DMA and PCI mapping from the available domheap pages. The
+        * chunk for DMA, PCI, etc., is a guestimate, as xen doesn't seem
+        * to have a good idea of what those requirements might be ahead
+        * of time, calculated at 1MB per 4GB of system memory */
+       domheap_pages = avail_domheap_pages();
+       p2m_pages = domheap_pages / PTRS_PER_PTE;
+       spare_hv_pages = domheap_pages / 4096;
+       max_dom0_size = (domheap_pages - (p2m_pages + spare_hv_pages))
+                        * PAGE_SIZE;
+       printk("Maximum permitted dom0 size: %luMB\n",
+              max_dom0_size / (1024*1024));
+
+       /* validate proposed dom0_size, fix up as needed */
+       if (dom0_size > max_dom0_size) {
+               printk("Reducing dom0 memory allocation from %luK to %luK "
+                      "to fit available memory\n",
+                      dom0_size / 1024, max_dom0_size / 1024);
+               dom0_size = max_dom0_size;
+       }
+
+       /* dom0_mem=0 can be passed in to give all available mem to dom0 */
+       if (dom0_size == 0) {
+               printk("Allocating all available memory to dom0\n");
+               dom0_size = max_dom0_size;
+       }
+
        /* Check dom0 size.  */
        if (dom0_size < 4 * 1024 * 1024) {
                panic("dom0_mem is too small, boot aborted"
@@ -1261,6 +1295,8 @@ int __init construct_dom0(struct domain 
        BUG_ON(v->is_initialised);
 
        printk("*** LOADING DOMAIN 0 ***\n");
+
+       calc_dom0_size();
 
        max_pages = dom0_size / PAGE_SIZE;
        d->max_pages = max_pages;
diff -r 039f2ccb1e38 xen/arch/ia64/xen/xensetup.c
--- a/xen/arch/ia64/xen/xensetup.c      Tue Jul 31 10:30:40 2007 -0600
+++ b/xen/arch/ia64/xen/xensetup.c      Wed Aug 01 13:44:31 2007 -0400
@@ -46,7 +46,6 @@ extern void early_setup_arch(char **);
 extern void early_setup_arch(char **);
 extern void late_setup_arch(char **);
 extern void hpsim_serial_init(void);
-extern void alloc_dom0(void);
 extern void setup_per_cpu_areas(void);
 extern void mem_init(void);
 extern void init_IRQ(void);
@@ -469,8 +468,6 @@ void __init start_kernel(void)
 
     trap_init();
 
-    alloc_dom0();
-
     init_xenheap_pages(__pa(xen_heap_start), xenheap_phys_end);
     printk("Xen heap: %luMB (%lukB)\n",
        (xenheap_phys_end-__pa(xen_heap_start)) >> 20,

Attachment: signature.asc
Description: OpenPGP digital signature

_______________________________________________
Xen-ia64-devel mailing list
Xen-ia64-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-ia64-devel