[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH v14 04/15] xen/percpu: defer freeing per-CPU areas after resume failures


  • To: Volodymyr Babchuk <Volodymyr_Babchuk@xxxxxxxx>
  • From: Mykola Kvach <xakep.amatop@xxxxxxxxx>
  • Date: Wed, 18 Nov 2026 12:22:35 +0200
  • Arc-authentication-results: i=1; mx.google.com; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=c1KAdteGyk2P92pTlx2NZGvjIuBeTZJzSA8OkPI18uI=; fh=DTLkauBK2JweO7DZXonqTvyBZL7CpVQ4HldXABfhCxU=; b=Eh+mAgv55g/bxyvzgiqSeNIZ6FUK1hrBkTW6Zd3wtvCN0jhfToCydusysRnVQXmLrs fizR4T/pFW8Y5fua/j1z5MX0xlvfUNTfXDiB8sv12rCMwjAJoQjbzzApSbWUN9lmulTv SxSFruAI8s0MxuCpXGKJ0SxA8oPABlQ9YW0OVbi+0ohx/FafQdqf13bGFrNjQvEpPhN1 xLh6DvZiWZaN6nU8Pujoxnl3q70bAd3CArVZnDmJd65Y70/QCDU+ImePEUxfWRmhKLDP pWiepNM0HObrHBXIZsxZj3rY9BEP0PNLn+/HAcbbANvAD6yk0eHyER29pXpw9AAcbCoB G3Ag==; darn=lists.xenproject.org
  • Arc-seal: i=1; a=rsa-sha256; t=1794997367; cv=none; d=google.com; s=arc-20260327; b=jCsyPp/WEmW5d0ufmQTrdDGqb6T//DIrkRweZ+B3A72fMf/qHwIaai4HDBJlKjcJ0+ BRCgu8xBvS2FcX9kiy7p1kvMCa9UIg8rcbO58CEEceBr7IPrprFHkPo4bP0C5tPTVHLu R/8DcLftwPvQD+arGwDowsGr+qMR4LnRsLKkvAcwvFqPIKBa2RF6swMCjJVTXuXI7PCe p6W4ZWteB4kHI8pqFdp0ViIxwM3A9L0KzoYWJltzGd9gdQYZw10hx4aCkt0H5bZRLoWL 18WMRbxSocz0QEvmZyHD9CRrCJ+VoUjjRWLxp44z5Rb4c5Cd82jVFvSJ1rMI3yHMo71M hCrQ==
  • Authentication-results: eu.smtp.expurgate.cloud; dkim=pass header.s=20260707 header.d=gmail.com header.i="@gmail.com" header.h="Content-Transfer-Encoding:Content-Type:Cc:To:Subject:Message-ID:Date:From:In-Reply-To:References:MIME-Version"
  • Cc: Mykola Kvach <Mykola_Kvach@xxxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxxx>, Andrew Cooper <andrew.cooper3@xxxxxxxxxx>, Anthony PERARD <anthony.perard@xxxxxxxxxx>, Michal Orzel <michal.orzel@xxxxxxx>, Jan Beulich <jbeulich@xxxxxxxx>, Julien Grall <julien@xxxxxxx>, Roger Pau Monné <roger@xxxxxxxxxxxxxx>, Stefano Stabellini <sstabellini@xxxxxxxxxx>, Juergen Gross <jgross@xxxxxxxx>
  • Delivery-date: Wed, 18 Nov 2026 10:22:55 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

Hi Volodymyr,

Thank you for the review.

On Wed, Nov 18, 2026 at 3:52 AM Volodymyr Babchuk
<Volodymyr_Babchuk@xxxxxxxx> wrote:
>
> Hi,
>
> CC'ed Juergen, because I have a question about cpupools.
>
> Mykola Kvach <Mykola_Kvach@xxxxxxxx> writes:
>
> > When a CPU fails to come back during system resume, cpu_up() sends
> > CPU_UP_CANCELED. After trying the other CPUs, enable_nonboot_cpus()
> > sends CPU_RESUME_FAILED for the same CPU.
> >
> > On architectures which do not park offline CPUs, both notifications
> > queue free_percpu_area(). The release metadata, including the rcu_head,
> > is stored in that CPU's per-CPU area. The first callback may already
> > have freed the area when the second notification arrives, or the same
> > rcu_head may be queued twice. The CPU_RESUME_FAILED callbacks also need
> > the area to clean up timers and CPU-pool state.
>
> A bit convoluted. As I see, problem is caused by the fact
> CPU_UP_CANCELED is a terminal state in a "simple" CPU up sequence, but
> is a transient state in a system resume sequence.

Agreed, I'll make this clearer in the description. If no other changes
are needed, could we update the commit message when applying the patch,
without rerolling the series?

> As such, theoretically
> other per_cpu callbacks might have a similar issue when they don't
> expect one more "CPU state is bad" callback. I can see that
> common/timer.c also tries to handle both callbacks as terminal
> ones. Probably, you need a similar fix there as well.

I checked timer.c. free_percpu_timers() sets ts->heap to dummy_heap
after freeing it, so the second call does not free the same allocation
again. The per-CPU area itself must remain valid, which this patch
ensures. I don't see a need for the same fix there.

>
> Actually, I am not sure why do we need CPU_RESUME_FAILED in the first
> place. As I can see, only cpupool handles it in a sensible way. Which
> begs a question: why cpupool can't call cpupool_cpu_remove_forced() on
> CPU_UP_CANCELED instead?

CPU_RESUME_FAILED runs after all CPUs have been tried, when the available
resources are known. This was the reason for adding it in commit
51c79e943fb3. Removing a CPU from its pool can require migrating vCPUs or
moving domains to pool0.

Also, if CPU_UP_PREPARE fails early, CPU_UP_CANCELED only unwinds part of
the notifier chain, so cpupool may not receive it. CPU_RESUME_FAILED runs
the full chain.

But let's wait for Juergen's input on the cpupool part.

Best regards,
Mykola


>
>
>
> >
> > Skip the per-CPU release for CPU_UP_CANCELED during resume. Let
> > CPU_RESUME_FAILED release the area after the other cleanup callbacks.
> >
> > This prepares the common code for Arm system suspend. x86 parks offline
> > CPUs, so the per-CPU notifier does not call free_percpu_area() for either
> > CPU_UP_CANCELED or CPU_RESUME_FAILED.
> >
> > Signed-off-by: Mykola Kvach <mykola_kvach@xxxxxxxx>
> > ---
> > Changes in V14:
> > - Clarify that x86 skips free_percpu_area() for CPU_UP_CANCELED and
> >   CPU_RESUME_FAILED.
> > - Remove the blank line after fallthrough. No functional change.
> >
> > Changes in v13:
> > - New patch.
> > ---
> >  xen/common/percpu.c | 8 ++++++++
> >  1 file changed, 8 insertions(+)
> >
> > diff --git a/xen/common/percpu.c b/xen/common/percpu.c
> > index f180f37253..757de111db 100644
> > --- a/xen/common/percpu.c
> > +++ b/xen/common/percpu.c
> > @@ -82,6 +82,14 @@ static int cf_check cpu_percpu_callback(
> >          break;
> >
> >      case CPU_UP_CANCELED:
> > +        /*
> > +         * On resume, CPU_RESUME_FAILED is sent after all CPUs have been
> > +         * tried. Keep the area alive for its cleanup callbacks and let
> > +         * that notification release it.
> > +         */
> > +        if ( system_state == SYS_STATE_resume )
> > +            break;
> > +        fallthrough;
> >      case CPU_DEAD:
> >      case CPU_RESUME_FAILED:
> >          if ( !park_offline_cpus && system_state != SYS_STATE_suspend )
>
> --
> WBR, Volodymyr



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.