RE: [EXT] Re: xen arm64 low power sleep support

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Thu, 31 Aug 2023, Anthony Chan wrote:
> On Thu, 30 Aug 2023, Stefano Stabellini wrote:
> > On Wed, 30 Aug 2023, Anthony Chan wrote:
> > > On Tue, 29 Aug 2023, Stefano Stabellini wrote:
> > > > On Tue, 29 Aug 2023, Anthony Chan wrote:
> > > > > Hi all,
> > > > >
> > > > > My name is Tony and I've been researching/developing using Xen
> > > > > for potential upcoming uses in our embedded systems.  I started
> > > > > with Xen using Xilinx tools about a year ago and still have lots
> > > > > to learn about what it can to do in the embedded space.  So far,
> > > > > I've managed to integrate Xen and Linux into an existing product
> > > > > that exclusively runs bare-metal code on a ZynqMP SoC and
> > > > > migrate some of the functionality into custom Linux driver/userspace.
> > > > >
> > > > > I'm now looking at low power support, for now at least between
> > > > > Xen
> > > > > (4.16) and Linux (5.15) dom0.  I've tried a few different Linux
> > > > > kernel configs around power management and each time I try to
> > > > > suspend from linux dom0 (via sysfs or systemctl), Xen will
> > > > > watchdog on
> > > > > dom0 guest.
> > > > > AFAIK, Xen should trap on a 'WFI' from guests, but from what I
> > > > > can tell debugging through the linux suspend process is it's
> > > > > spinning in a 'suspend- to-idle' loop before it can get to
> > > > > issuing a 'WFI' or using PSCI interface to notify Xen.  I'm
> > > > > beginning to suspect that 'low power' support for embedded arm64
> > > > > just isn't quite there yet, or am I missing something in the configs?
> > > > >
> > > > > I realize this could very well be a Linux 'issue' but checking
> > > > > here first.  I know Xen presents a flattened device tree to
> > > > > Linux without CPU idle-state nodes and maybe this is causing the
> > > > > linux guest to only do the suspend- to-idle mode?  I should
> > > > > mention that I'm booting up using dom0less feature if that
> > > > > matters.
> > > >
> > > >
> > > > Hi Anthony,
> > > >
> > > > Assuming you are using the default Xen command line parameters for
> > > > Xilinx boards: sched=null vwfi=native, then if the guest uses WFI,
> > > > the CPU will execute WFI directly and go into low power mode.
> > > Yes, using these command line params.
> > >
> > > > Given the issue you are describing, I am suspecting the guest is
> > > > not issuing
> > > > WFI: that is simple and known to work. Instead, I suspect that
> > > > Linux might be trying to use PSCI_suspend in a way that is not
> > > > supported or well- implemented by Xen.
> > > >
> > > > Can you check? You can add a printk in Linux
> > > > drivers/firmware/psci/psci.c:__psci_cpu_suspend or in Xen
> > > > xen/arch/arm/vpsci.c:do_psci_0_2_cpu_suspend
> > > Instrumented both places it doesn't appear to reach there.  In
> > > kernel/power/suspend.c, there's a call to s2idle_loop that it's currently 
> > > 'stuck'
> > > in and I think it doesn't get to the psci suspend your referring
> > > till afterwards, when suspend_ops->enter is called.  Unfortunately,
> > > without any idle-states nodes in the FDT, the only suspend state
> > > Linux is defaults to is 'suspend to idle'.
> >
> > The fact that Linux uses "suspend to idle" is not a problem because as
> > I mentioned WFI or PSCI_suspent are not different on Xen. That part is OK.
> What if using "suspend to idle" is preventing a WFI/PSCI_suspend?  Which is
> what I believe I'm currently seeing in my setup.  In kernel/power/suspend.c,
> suspend_devices_and_enter(), it gets into the this s2idle_loop and upon
> resuming from idle, it jumps past the point where I believe a
> WFI/PSCI_suspend can happen.
> if (state == PM_SUSPEND_TO_IDLE) {
>                 s2idle_loop();
>                 goto Platform_wake;
> }
 
If that is the case, then it looks like a Linux bug. Maybe something
along these lines?

https://lore.kernel.org/linux-arm-kernel/4665489.GXAFRqVoOG@kreacher/T/#m6edda92d0b5dc09f8e05e7d6db3807501b7249f4