Re: [Xen-ia64-devel] [PATCH] Use saner dom0 memory and vcpu defaults, don't panic on over-allocation

[Jarod Wilson <jwilson@xxxxxxxxxx>]

Alex Williamson wrote:
> On Fri, 2007-08-03 at 11:09 -0400, Jarod Wilson wrote:
> > > > > Indeed, on my 16GB system, only 8MB less from the v2 incantation, and
> > > > > the dom0_mem=0 option does properly allocate all available memory to
> > > > > dom0. I'm quite happy with this version if everyone else is...
> > > > Eep! I retract my happiness... Seems with all memory allocated like
> > > > this, I can't get a guest to actually boot. I get this when I try to
> > > > bring one up via xm create:
>    It's still not completely happy on my 96G system.  If I specify
> dom0_mem=0, dom0 fails to startup.  It's incredibly close to the right
> value though.  The max calculated memory is:
>
> (XEN) Maximum permitted dom0 size: 97841MB
>
> This fails just before launching dom0 with this:
>
> (XEN) Domain0 EFI passthrough: ACPI 2.0=0x3fdda000 SMBIOS=0x3e52a000
> (XEN) assign_new_domain_page: Can't alloc!!!! Aaaargh!
> (XEN) 
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) assign_new_domain0_page: can't allocate page for 
> dom0****************************************
>
> If I reduce this slightly and boot with dom0_mem=97840MB, dom0 starts to
> boot, then hits this:
>
> CPU 1: synchronized ITC with CPU 0 (last diff -2 cycles, maxerr 154 cycles)
> CPU 2: synchronized ITC with CPU 0 (last diff -8 cycles, maxerr 159 cycles)
> CPU 3: synchronized ITC with CPU 0 (last diff -7 cycles, maxerr 159 cycles)
> Brought up 4 CPUs
> Total of 4 processors activated (12753.30 BogoMIPS).
> migration_cost=11839
> (XEN) mm.c:520:d0 Warning: UC to WB for mpaddr=3e52a010
> DMI 2.3 present.
> NET: Registered protocol family 16
> ACPI: bus type pci registered
> ACPI: Interpreter enabled
> ACPI: Using IOSAPIC for interrupt routing
> ACPI: PCI Root Bridge [L000] (0000:00)
> (XEN) __assign_domain_page:871 WARNING can't assign page domain 
> 0xf000000007c18080 id 0
> (XEN)   already assigned pte_val 0x6400000000000000
> (XEN)   mpaddr 0x00000800bc000000 physaddr 0x00000800bc000000 flags 0x2
> (XEN) __assign_domain_page:871 WARNING can't assign page domain 
> 0xf000000007c18080 id 0
> (XEN)   already assigned pte_val 0xf000000000000000
> (XEN)   mpaddr 0x00000800bc004000 physaddr 0x00000800bc004000 flags 0x2
> (XEN) __assign_domain_page:871 WARNING can't assign page domain 
> 0xf000000007c18080 id 0
> (XEN)   already assigned pte_val 0xf000ff54f0000000
> (XEN)   mpaddr 0x00000800bc008000 physaddr 0x00000800bc008000 flags 0x2
> ...
>
> To get it to boot all the way to userspace, I had to reduce dom0's
> memory a full 16MB further, down to dom0_mem=97824M.  Each few MB more
> kept for Xen allowed the boot to enumerate another PCI root bridge.  I
> suspect it's the ioremaps going on there that are eating up free heap
> pages.  Maybe this also suggests that xen doesn't really know how much
> dynamic memory it might take to boot dom0 all the way.  Thanks,
After poking at the x86 code again, it looks like there's simply a guess 
there too for extra memory to reserve. From xen/arch/x86/domain_build.c:
    /*
     * If domain 0 allocation isn't specified, reserve 1/16th of available
     * memory for things like DMA buffers. This reservation is clamped to
     * a maximum of 128MB.
     */
    if ( dom0_nrpages == 0 )
    {
        dom0_nrpages = avail;
        dom0_nrpages = min(dom0_nrpages / 16, 128L << (20 - PAGE_SHIFT));
        dom0_nrpages = -dom0_nrpages;
    }


Based on empirical testing, it seems something along the lines of 1MB 
per 4GB of RAM, capped at like 64MB (though probably not necessary), 
might be sane. Its not exact of course, since memory amount doesn't 
necessarily indicate how many PCI root bridges there are to allocate, 
etc., but it seems to be a reasonable guess... The attached is a crack 
at that, and on your 96G machine, should decrease the default "all" 
memory allocation by about 24MB, allowing it to fully boot.
Here's another thought: we could play it extra-safe, and slide that 
allocation down another 12-24MB, and possibly even switch to allocating 
"all" memory to dom0 by default... Then allocate all cpus too, and we'd 
have parity with x86 here... :)
--
Jarod Wilson
jwilson@xxxxxxxxxx

Some ia64 xen dom0 tweaks:
* Increase default memory allocation from 512M to 4G
* Increase default vcpu allocation from 1 to 4
* If need be, scale down requested memory allocation to fit
  available memory, rather than simply panicking
* If dom0_mem=0 is specified, allocate all available mem

Signed-off-by: Jarod Wilson <jwilson@xxxxxxxxxx>

diff -r 039f2ccb1e38 xen/arch/ia64/xen/domain.c
--- a/xen/arch/ia64/xen/domain.c        Tue Jul 31 10:30:40 2007 -0600
+++ b/xen/arch/ia64/xen/domain.c        Sat Aug 04 21:50:56 2007 -0400
@@ -52,10 +52,11 @@
 #include <asm/perfmon.h>
 #include <public/vcpu.h>
 
-static unsigned long __initdata dom0_size = 512*1024*1024;
+/* dom0_size: default memory allocation for dom0 (~4GB) */
+static unsigned long __initdata dom0_size = 4096UL*1024UL*1024UL;
 
 /* dom0_max_vcpus: maximum number of VCPUs to create for dom0.  */
-static unsigned int __initdata dom0_max_vcpus = 1;
+static unsigned int __initdata dom0_max_vcpus = 4;
 integer_param("dom0_max_vcpus", dom0_max_vcpus); 
 
 extern char dom0_command_line[];
@@ -1195,8 +1196,42 @@ static void __init loaddomainelfimage(st
        }
 }
 
-void __init alloc_dom0(void)
-{
+static void __init calc_dom0_size(void)
+{
+       unsigned long domheap_pages;
+       unsigned long p2m_pages;
+       unsigned long spare_hv_pages;
+       unsigned long max_dom0_size;
+
+       /* Estimate maximum memory we can safely allocate for dom0
+        * by subtracting the p2m table allocation and a chunk of memory
+        * for DMA and PCI mapping from the available domheap pages. The
+        * chunk for DMA, PCI, etc., is a guestimate, as xen doesn't seem
+        * to have a good idea of what those requirements might be ahead
+        * of time, calculated at 1MB per 4GB of system memory, but capped
+        * at 64MB of RAM, based on emperical testing. */
+       domheap_pages = avail_domheap_pages();
+       p2m_pages = domheap_pages / PTRS_PER_PTE;
+       spare_hv_pages = min(domheap_pages / 4096, 64UL << (20 - PAGE_SHIFT));
+       max_dom0_size = (domheap_pages - (p2m_pages + spare_hv_pages))
+                        * PAGE_SIZE;
+       printk("Maximum permitted dom0 size: %luMB\n",
+              max_dom0_size / (1024*1024));
+
+       /* validate proposed dom0_size, fix up as needed */
+       if (dom0_size > max_dom0_size) {
+               printk("Reducing dom0 memory allocation from %luK to %luK "
+                      "to fit available memory\n",
+                      dom0_size / 1024, max_dom0_size / 1024);
+               dom0_size = max_dom0_size;
+       }
+
+       /* dom0_mem=0 can be passed in to give all available mem to dom0 */
+       if (dom0_size == 0) {
+               printk("Allocating all available memory to dom0\n");
+               dom0_size = max_dom0_size;
+       }
+
        /* Check dom0 size.  */
        if (dom0_size < 4 * 1024 * 1024) {
                panic("dom0_mem is too small, boot aborted"
@@ -1261,6 +1296,8 @@ int __init construct_dom0(struct domain 
        BUG_ON(v->is_initialised);
 
        printk("*** LOADING DOMAIN 0 ***\n");
+
+       calc_dom0_size();
 
        max_pages = dom0_size / PAGE_SIZE;
        d->max_pages = max_pages;
diff -r 039f2ccb1e38 xen/arch/ia64/xen/xensetup.c
--- a/xen/arch/ia64/xen/xensetup.c      Tue Jul 31 10:30:40 2007 -0600
+++ b/xen/arch/ia64/xen/xensetup.c      Wed Aug 01 13:44:31 2007 -0400
@@ -46,7 +46,6 @@ extern void early_setup_arch(char **);
 extern void early_setup_arch(char **);
 extern void late_setup_arch(char **);
 extern void hpsim_serial_init(void);
-extern void alloc_dom0(void);
 extern void setup_per_cpu_areas(void);
 extern void mem_init(void);
 extern void init_IRQ(void);
@@ -469,8 +468,6 @@ void __init start_kernel(void)
 
     trap_init();
 
-    alloc_dom0();
-
     init_xenheap_pages(__pa(xen_heap_start), xenheap_phys_end);
     printk("Xen heap: %luMB (%lukB)\n",
        (xenheap_phys_end-__pa(xen_heap_start)) >> 20,

_______________________________________________
Xen-ia64-devel mailing list
Xen-ia64-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-ia64-devel