Re: [Xen-devel] RFC: QEMU bumping memory limit and domain restore

[Don Slutz <don.slutz@xxxxxxxxx>]

On 06/03/15 09:53, George Dunlap wrote:
> On 06/03/2015 02:32 PM, Andrew Cooper wrote:
>> On 03/06/15 14:22, George Dunlap wrote:
>>> On Tue, Jun 2, 2015 at 3:05 PM, Wei Liu <wei.liu2@xxxxxxxxxx> wrote:
>>>> Previous discussion at [0].
>>>>
>>>> For the benefit of discussion, we refer to max_memkb inside hypervisor
>>>> as hv_max_memkb (name subject to improvement). That's the maximum number
>>>> of memory a domain can use.
>>> Why don't we try to use "memory" for virtual RAM that we report to the
>>> guest, and "pages" for what exists inside the hypervisor?  "Pages" is
>>> the term the hypervisor itself uses internally (i.e., set_max_mem()
>>> actually changes a domain's max_pages value).
>>>
>>> So in this case both guest memory and option roms are created using
>>> hypervisor pages.
>>>
>>>> Libxl doesn't know hv_max_memkb for a domain needs prior to QEMU start-up
>>>> because of optional ROMs etc.
>>> So a translation of this using "memory/pages" terminology would be:
>>>
>>> QEMU may need extra pages from Xen to implement option ROMS, and so at
>>> the moment it calls set_max_mem() to increase max_pages so that it can
>>> allocate more pages to the guest.  libxl doesn't know what max_pages a
>>> domain needs prior to qemu start-up.
>>>
>>>> Libxl doesn't know the hv_max_memkb even after QEMU start-up, because
>>>> there is no mechanism to community between QEMU and libxl. This is an
>>>> area that needs improvement, we've encountered problems in this area
>>>> before.
>>> [translating]
>>> Libxl doesn't know max_pages  even after qemu start-up, because there
>>> is no mechanism to communicate between qemu and libxl.

This is not 100% true.  libxl and QEMU do communicate.  Currently QEMU
does not record it's changes but I have not search all the info you can
get.  Also libxl "knows" what it set maxmem to before QEMU starts and it
can read (via xc) what it is after QEMU is done all the changes.  QEMU
only responds to commands from libxl at this time.

>>>> QEMU calls xc_domain_setmaxmem to increase hv_max_memkb by N pages. Those
>>>> pages are only accounted in hypervisor. During migration, libxl
>>>> (currently) doesn't extract that value from hypervisor.
>>> [translating]
>>> qemu calls xc_domain_setmaxmem to increase max_pages by N pages.
>>> Those pages are only accounted for in the hypervisor.  libxl
>>> (currently) does not extract that value from the hypervisor.

Yes.

>>>> So now the problem is on the remote end:
>>>>
>>>> 1. Libxl indicates domain needs X pages.
>>>> 2. Domain actually needs X + N pages.
>>>> 3. Remote end tries to write N more pages and fail.
>>>>
>>>> This behaviour currently doesn't affect normal migration (that you
>>>> transfer libxl JSON to remote, construct a domain, then start QEMU)
>>>> because QEMU won't bump hv_max_memkb again. This is by design and
>>>> reflected in QEMU code.
>>> I don't understand this paragraph -- does the remote domain actually
>>> need X+N pages or not?  If it does, in what way does this behavior
>>> "not affect normal migration"?

As far as I know is does.  And it affects "xl save" and "xl restore".

However you need more then 4 e1000 nics (for example) to see this.

>>>> This behaviour affects COLO and becomes a bug in that case, because
>>>> secondary VM's QEMU doesn't go through the same start-of-day
>>>> initialisation (Hongyang, correct me if I'm wrong), i.e. no bumping
>>>> hv_max_memkb inside QEMU.
>>>>
>>>> Andrew plans to embed JSON inside migration v2 and COLO is based on
>>>> migration v2. The bug is fixed if JSON is correct in the first place.
>>>>
>>>> As COLO is not yet upstream, so this bug is not a blocker for 4.6. But
>>>> it should be fixed for the benefit of COLO.
>>>>
>>>> So here is a proof of concept patch to record and honour that value
>>>> during migration.  A new field is added in IDL. Note that we don't
>>>> provide xl level config option for it and mandate it to be default value
>>>> during domain creation. This is to prevent libxl user from using it to
>>>> avoid unforeseen repercussions.
>>>>
>>>> This patch is compiled test only. If we agree this is the way to go I
>>>> will test and submit a proper patch.
>>> Reading max_pages from Xen and setting it on the far side seems like a
>>> reasonable option.
>> It is the wrong level to fix the bug.  Yes - it will (and does) fix one
>> manifestation of the bug, but does not solve the problem.
>>
>>>   Is there a reason we can't add a magic XC_SAVE_ID
>>> for v1, like we do for other parameters?

Wei Liu failed to refer to:

Date: Tue, 14 Apr 2015 18:06:26 -0400
Subject: [PATCH v2 1/1] tools: Handle xc_maxmem adjustments
Thread-Topic: [PATCH v2 1/1] tools: Handle xc_maxmem adjustments
Thread-Index: AdB2/0pJoAZieMWqTomVEBNbV7iiEA==
Message-ID: <1429049186-27343-1-git-send-email-dslutz@xxxxxxxxxxx>
References: <20150414175437.GB11717@xxxxxxxxxxxxxxxxxxxxx>
In-Reply-To: <20150414175437.GB11717@xxxxxxxxxxxxxxxxxxxxx>

Which was this coded up.

>> Amongst other things, playing with xc_domain_setmaxmem() is liable to
>> cause a PoD domain to be shot by Xen because the PoD cache was not
>> adjusted at the same time that maxmem was.
> As far as I can tell that's completely unrelated.
>
> The PoD target needs to be updated when you change the *balloon driver*
> target in xenstore.  libxl_domain_setmaxmem() for instance won't update
> the PoD target either.
>

And xc_maxmem does not change xenstore.

>> Only libxl is in a position to safely adjust domain memory.
> I was initially of the same mind as you in this matter.  But I think if
> you stop thinking about this as "the domain's memory", and start instead
> thinking about it as qemu modifying the state of the machine (by, say,
> adding option roms), then I think it can be less strict.  The *memory*
> (i.e., what is seen by the guest as virtual RAM) is controlled by libxl,
> but the *pages* may be manipulated by others (as they are by the balloon
> driver, for instance).
>
> Although actually -- I guess that exposes another issue with this: what
> if someone calls setmaxmem in libxl?  libxl doesn't know about the extra
> N pages that qemu wanted, so it won't factor that in to its setmaxmem
> calculation.  In particular, qemu's read-modify-write of setmaxmem isn't
> idempotent -- there's almost certainly a race if someone were to call
> libxl_domain_setmaxmem at the same time qemu was trying to add a few pages.

There have been changes in the area.  libxl_setmem now reads, modifies
and writes.

> In any case, as Wei said, you have the problem that by the time qemu
> figures out that it wants some extra pages to implement an option rom,
> libxl has already determined that its job was complete and walked off.

Not in my testing.  libxl wants to get the VNC "display" that QEMU
configured which happens after QEMU has changed maxmem.

> Changing that requires a more architectural change.
>
> Perhaps joining this with the security deprivileging thing would make
> some sense?  I.e., have qemu do no hypercalls at all, but have them done
> instead by a "babysitter" process from libxl, which could then update
> the domain config state as well.

Not sure.
    -Don Slutz

>  -George
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel