[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [PATCH] Revert "domctl: improve locking during domain destruction"

  • To: Jan Beulich <jbeulich@xxxxxxxx>, Julien Grall <julien@xxxxxxx>
  • From: Jim Fehlig <jfehlig@xxxxxxxx>
  • Date: Thu, 26 Mar 2020 10:55:52 -0600
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=suse.com; dmarc=pass action=none header.from=suse.com; dkim=pass header.d=suse.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ySCbFiJD3Ir5r8VAj6JmRO4mZAAlzbOlH14SOa1Fj9g=; b=BPrOhKgaKy+ON5Xr0Lbfh3e35dsGovcjpoH95veW+/ezdbClf+EyokvNMJzQZ74CBfHxVlwzOMAjOmQD9vWnTOQPvMYIwB0WqpadmVgS7B6HnfkJAmjyZ7Ckv4pOEmqM7CrQ4jzuryjP/sJSxZEYVTtgdEV3EKsQxvCqCq89C72QgNGZzLaGqf1g7JPRI79SQK/yo4f6uLYmmOMc/vVWzOA8e/mkFlM7WwDHcD6Wa3Jn8MIsyvVlHTsQOuF+qWt7MzY1CYbZ60VUqqwot23gWJEmCmBsoas09pVXG7KP5CGHy23d6Se2KRNoBv4K+owAT+LcbP4NfRDpFj+ziRJHMA==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HWoy7LA7nVQ+Qrh+WhrkkkTQI1gdO5JX4d1hDUW2mfQTg7EExpdY8AVV/l+hLQklOq/4O03nOPhyo0+34PTT0qCMmIGUyIjuD8JUky3B/0GlMCc5nH5KvEE8Zyd7Rs/npsj2Hus1knl8c/N+pmadyLF1gDQmVD/aa5WCp0Z7tum/OY0CBfuP7STm6xOqGDbX4LS7G7gdKGoayV70jXhXnIjU641vninGwg+ZIXycG4+m/oeQKN77uP73ZjFU2FrqUCymiWBlfrLaNVVNBoP3hpK2XWN6Z4GGvrlJF2X0DncsHV2Z5aPOCwVZvP16Y9OnItSGGfXE0bfRmyrNcZkFoQ==
  • Authentication-results: spf=none (sender IP is ) smtp.mailfrom=jfehlig@xxxxxxxx;
  • Cc: Hongyan Xia <hx242@xxxxxxx>, Charles Arnold <CARNOLD@xxxxxxxx>, Stefano Stabellini <sstabellini@xxxxxxxxxx>, Wei Liu <wl@xxxxxxx>, Andrew Cooper <andrew.cooper3@xxxxxxxxxx>, Ian Jackson <ian.jackson@xxxxxxxxxxxxx>, George Dunlap <george.dunlap@xxxxxxxxxx>, xen-devel@xxxxxxxxxxxxxxxxxxxx
  • Delivery-date: Thu, 26 Mar 2020 17:03:40 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On 3/25/20 1:11 AM, Jan Beulich wrote:
On 24.03.2020 19:39, Julien Grall wrote:
On 24/03/2020 16:13, Jan Beulich wrote:
On 24.03.2020 16:21, Hongyan Xia wrote:
From: Hongyan Xia <hongyxia@xxxxxxxxxx>
In contrast,
after dropping that commit, parallel domain destructions will just fail
to take the domctl lock, creating a hypercall continuation and backing
off immediately, allowing the thread that holds the lock to destroy a
domain much more quickly and allowing backed-off threads to process
events and irqs.

On a 144-core server with 4TiB of memory, destroying 32 guests (each
with 4 vcpus and 122GiB memory) simultaneously takes:

before the revert: 29 minutes
after the revert: 6 minutes

This wants comparing against numbers demonstrating the bad effects of
the global domctl lock. Iirc they were quite a bit higher than 6 min,
perhaps depending on guest properties.

Your original commit message doesn't contain any clue in which
cases the domctl lock was an issue. So please provide information
on the setups you think it will make it worse.

I did never observe the issue myself - let's see whether one of the SUSE
people possibly involved in this back then recall (or have further
pointers; Jim, Charles?), or whether any of the (partly former) Citrix
folks do. My vague recollection is that the issue was the tool stack as
a whole stalling for far too long in particular when destroying very
large guests.

I too only have a vague memory of the issue but do recall shutting down large guests (e.g. 500GB) taking a long time and blocking other toolstack operations. I haven't checked on the behavior in quite some time though.

One important aspect not discussed in the commit message
at all is that holding the domctl lock block basically _all_ tool stack
operations (including e.g. creation of new guests), whereas the new
issue attempted to be addressed is limited to just domain cleanup.

I more vaguely recall shutting down the host taking a *long* time when dom0 had large amounts of memory, e.g. when it had all host memory (no dom0_mem= setting and autoballooning enabled).




Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.