Re: [Xen-devel] [Qemu-devel] [PATCH] increase maxmem before calling xc_domain_populate_physmap

[Don Slutz <dslutz@xxxxxxxxxxx>]

On 12/02/14 07:26, Stefano Stabellini wrote:
> On Mon, 1 Dec 2014, Don Slutz wrote:
> > On 12/01/14 10:37, Stefano Stabellini wrote:
> > > On Mon, 1 Dec 2014, Don Slutz wrote:
> > > > On 11/27/14 05:48, Stefano Stabellini wrote:
> > [...]
> >
> > > > Works fine in both claim modes and with PoD used (maxmem > memory).  Do
> > > > not know how to test with tmem.  I do not see how it would be worse then
> > > > current
> > > > code that does not auto increase.  I.E. even without a xen change, I think
> > > > something
> > > > like this could be done.
> > > OK, good to know. I am OK with increasing maxmem only if it is strictly
> > > necessary.
> > >
> > >
> > > > > > My testing shows a free 32 pages that I am not sure where they come
> > > > > > from.
> > > > > > But
> > > > > > the code about is passing my 8 nics of e1000.
> > > > > I think that raising maxmem a bit higher than necessary is not too bad.
> > > > > If we really care about it, we could lower the limit after QEMU's
> > > > > initialization is completed.
> > > > Ok.  I did find the 32 it is VGA_HOLE_SIZE.  So here is what I have which
> > > > includes
> > > > a lot of extra printf.
> > > In QEMU I would prefer not to assume that libxl increased maxmem for the
> > > vga hole. I would rather call xc_domain_setmaxmem twice for the vga hole
> > > than tie QEMU to a particular maxmem allocation scheme in libxl.
> > Ok.  The area we are talking about is 0x000a0000 to 0x000c0000.
> > It is in libxc (xc_hvm_build_x86), not libxl.   I have no issue with a name
> > change to
> > some thing like QEMU_EXTRA_FREE_PAGES.
> Sorry, I was thinking about the relocated videoram region, the one
> allocated by xen_add_to_physmap in QEMU.
>
> Regarding 0x000a0000-0x000c0000, according to this comment:
>
>      /*
>       * Allocate memory for HVM guest, skipping VGA hole 0xA0000-0xC0000.
>       *
>
> xc_hvm_build_x86 shouldn't be allocating memory for it.
> In fact if I remember correctly 0xa0000-0xc0000 is trapped and emulated.
Clearly I am not writing clear enough statements.

xc_hvm_build_x86 is not allocating memory for it.  libxl__build_pre() sets
maxmem via xc_domain_setmaxmem().  Then xc_hvm_build_x86 allocates
memory skipping the VGA hole 0xA0000-0xC0000. So difference between
maxmem (max_pages, xlinfo->max_memkb) and tot_pages 
(xlinfo->current_memkb) is videoram + LIBXL_MAXMEM_CONSTANT + 32
(I.E. what VGA_HOLE_SIZE is definded as).


> > My testing has shows that some of these 32 pages are used outside of QEMU.
> > I am seeing just 23 free pages using a standalone program to display
> > the same info after a CentOS 6.3 guest is done booting.
> >
> > > In libxl I would like to avoid increasing mamxem for anything QEMU will
> > > allocate later, that includes rom and vga vram. I am not sure how to
> > > make that work with older QEMU versions that don't call
> > > xc_domain_setmaxmem by themselves yet though. Maybe we could check the
> > > specific QEMU version in libxl and decide based on that. Or we could
> > > export a feature flag in QEMU.
> > Yes, it would be nice to adjust libxl to not increase maxmem. However since
> > videoram is included in memory (and maxmem), making the change related
> > to vram is a bigger issue.
> >
> > the rom change is much simpler.
> >
> > Currently I do not know of a way to do different things based on the QEMU
> > version
> > and/or features (this includes getting the QEMU version in libxl).
> >
> > I have been going with:
> > 1) change QEMU 1st.
> > 2) Wait for an upstream version of QEMU with this.
> > 3) change xen to optionally use a feature in the latest QEMU.
> Let's try to take this approach for the videoram too
Ok.

> > > > --- a/xen-hvm.c
> > > > +++ b/xen-hvm.c
> > > > @@ -67,6 +67,7 @@ static inline ioreq_t *xen_vcpu_ioreq(shared_iopage_t
> > > > *shared_page, int vcpu)
> > > >    #endif
> > > >
> > > >    #define BUFFER_IO_MAX_DELAY  100
> > > > +#define VGA_HOLE_SIZE (0x20)
> > > >
> > > >    typedef struct XenPhysmap {
> > > >        hwaddr start_addr;
> > > > @@ -219,6 +220,11 @@ void xen_ram_alloc(ram_addr_t ram_addr, ram_addr_t
> > > > size,
> > > > MemoryRegion *mr)
> > > >        xen_pfn_t *pfn_list;
> > > >        int i;
> > > >        xc_dominfo_t info;
> > > > +    unsigned long max_pages, free_pages, real_free;
> > > > +    long need_pages;
> > > > +    uint64_t tot_pages, pod_cache_pages, pod_entries;
> > > > +
> > > > +    trace_xen_ram_alloc(ram_addr, size, mr->name);
> > > >
> > > >        if (runstate_check(RUN_STATE_INMIGRATE)) {
> > > >            /* RAM already populated in Xen */
> > > > @@ -232,13 +238,6 @@ void xen_ram_alloc(ram_addr_t ram_addr, ram_addr_t
> > > > size,
> > > > MemoryRegion *mr)
> > > >            return;
> > > >        }
> > > >
> > > > -    fprintf(stderr, "%s: alloc "RAM_ADDR_FMT
> > > > -            " bytes (%ld Kib) of ram at "RAM_ADDR_FMT
> > > > -            " mr.name=%s\n",
> > > > -            __func__, size, (long)(size>>10), ram_addr, mr->name);
> > > > -
> > > > -    trace_xen_ram_alloc(ram_addr, size);
> > > > -
> > > >        nr_pfn = size >> TARGET_PAGE_BITS;
> > > >        pfn_list = g_malloc(sizeof (*pfn_list) * nr_pfn);
> > > >
> > > > @@ -246,11 +245,38 @@ void xen_ram_alloc(ram_addr_t ram_addr, ram_addr_t
> > > > size,
> > > > MemoryRegion *mr)
> > > >            pfn_list[i] = (ram_addr >> TARGET_PAGE_BITS) + i;
> > > >        }
> > > >
> > > > -    if (xc_domain_getinfo(xen_xc, xen_domid, 1, &info) < 0) {
> > > > +    if ((xc_domain_getinfo(xen_xc, xen_domid, 1, &info) != 1) ||
> I think it's best to call xc_domain_getinfolist.
Will switch.

> > > > +       (info.domid != xen_domid)) {
> > > >            hw_error("xc_domain_getinfo failed");
> > > >        }
> > > > -    if (xc_domain_setmaxmem(xen_xc, xen_domid, info.max_memkb +
> > > > -                            (nr_pfn * XC_PAGE_SIZE / 1024)) < 0) {
> > > > +    max_pages = info.max_memkb * 1024 / XC_PAGE_SIZE;
> > > > +    free_pages = max_pages - info.nr_pages;
> > > > +    real_free = free_pages;
> > > > +    if (free_pages > VGA_HOLE_SIZE) {
> > > > +        free_pages -= VGA_HOLE_SIZE;
> > > > +    } else {
> > > > +        free_pages = 0;
> > > > +    }
> I don't think we need to subtract VGA_HOLE_SIZE.
If you do not use some slack (like 32 here), xen does report:


(d5) [2014-11-29 17:07:21] Loaded SeaBIOS
(d5) [2014-11-29 17:07:21] Creating MP tables ...
(d5) [2014-11-29 17:07:21] Loading ACPI ...
(XEN) [2014-11-29 17:07:21] page_alloc.c:1568:d5 Over-allocation for 
domain 5: 1057417 > 1057416
(XEN) [2014-11-29 17:07:21] memory.c:158:d5 Could not allocate order=0 
extent: id=5 memflags=0 (0 of 1)
(d5) [2014-11-29 17:07:21] vm86 TSS at 00098c00
(d5) [2014-11-29 17:07:21] BIOS map:


However while QEMU startup ends with 32 "free" pages (maxmem - curmem)
this quickly changes to 23.  I.E. there are 7 more places that act like 
a call
to xc_domain_populate_physmap_exact() but do not log errors if there is
no free pages.  And just to make sure I do not fully understand what is
happening here, after the message above, the domain appears to work
fine and ends up with 8 "free" pages (code I do not know about ends up
releasing populated pages).

So my current thinking is to have QEMU leave a few (9 to 32 (64?)) pages 
"free".

> > > > +    need_pages = nr_pfn - free_pages;
> > > > +    fprintf(stderr, "%s: alloc "RAM_ADDR_FMT
> > > > +            " bytes (%ld KiB) of ram at "RAM_ADDR_FMT
> > > > +            " mp=%ld fp=%ld nr=%ld need=%ld mr.name=%s\n",
> > > > +            __func__, size, (long)(size>>10), ram_addr,
> > > > +           max_pages, free_pages, nr_pfn, need_pages,
> > > > +            mr->name);
> > > > +    if (xc_domain_get_pod_target(xen_xc, xen_domid, &tot_pages,
> > > > +                                 &pod_cache_pages, &pod_entries) >= 0) {
> > > > +        unsigned long populated = tot_pages - pod_cache_pages;
> > > > +        long delta_tot = tot_pages - info.nr_pages;
> > > > +
> > > > +        fprintf(stderr, "%s: PoD pop=%ld tot=%ld(%ld) cnt=%ld ent=%ld
> > > > nop=%ld
> > > > free=%ld\n",
> > > > +                __func__, populated, (long)tot_pages, delta_tot,
> > > > +                (long)pod_cache_pages, (long)pod_entries,
> > > > +               (long)info.nr_outstanding_pages, real_free);
> > > > +    }
> > > What is the purpose of calling xc_domain_get_pod_target here? It doesn't
> > > look like is affecting the parameters passed to xc_domain_setmaxmem.
> > This was just to test and see if anything was needed for PoD mode.
> > I did not see anything.
> >
> > Did you want me to make a patch to send to the list that has my proposed
> > changes?
> Yep, that's fine.
>
>
> > I do think that what I have would be fine to do since most of the time the
> > new code does nothing with the current xen (and older versions), until
> > more then 4 nics are configured for xen.
> >
> > It would be good to have the change:
> >
> > [PATCH] libxl_set_memory_target: retain the same maxmem offset on top of the
> > current target
> >
> > (When working) in xen before using a QEMU with this change.
> >
> > Not sure I would push for 2.2 however.
> I wouldn't.
Will mark as for 2.3

   -Don Slutz



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel