RE: [EXT] Re: xen arm64 low power sleep support

[Anthony Chan <anthonychan@xxxxxxxxxx>]

On Thu, 30 Aug 2023, Stefano Stabellini wrote:
> On Wed, 30 Aug 2023, Anthony Chan wrote:
> > On Tue, 29 Aug 2023, Stefano Stabellini wrote:
> > > On Tue, 29 Aug 2023, Anthony Chan wrote:
> > > > Hi all,
> > > >
> > > > My name is Tony and I've been researching/developing using Xen
> > > > for potential upcoming uses in our embedded systems.  I started
> > > > with Xen using Xilinx tools about a year ago and still have lots
> > > > to learn about what it can to do in the embedded space.  So far,
> > > > I've managed to integrate Xen and Linux into an existing product
> > > > that exclusively runs bare-metal code on a ZynqMP SoC and
> > > > migrate some of the functionality into custom Linux driver/userspace.
> > > >
> > > > I'm now looking at low power support, for now at least between
> > > > Xen
> > > > (4.16) and Linux (5.15) dom0.  I've tried a few different Linux
> > > > kernel configs around power management and each time I try to
> > > > suspend from linux dom0 (via sysfs or systemctl), Xen will
> > > > watchdog on
> > > > dom0 guest.
> > > > AFAIK, Xen should trap on a 'WFI' from guests, but from what I
> > > > can tell debugging through the linux suspend process is it's
> > > > spinning in a 'suspend- to-idle' loop before it can get to
> > > > issuing a 'WFI' or using PSCI interface to notify Xen.  I'm
> > > > beginning to suspect that 'low power' support for embedded arm64
> > > > just isn't quite there yet, or am I missing something in the configs?
> > > >
> > > > I realize this could very well be a Linux 'issue' but checking
> > > > here first.  I know Xen presents a flattened device tree to
> > > > Linux without CPU idle-state nodes and maybe this is causing the
> > > > linux guest to only do the suspend- to-idle mode?  I should
> > > > mention that I'm booting up using dom0less feature if that
> > > > matters.
> > >
> > >
> > > Hi Anthony,
> > >
> > > Assuming you are using the default Xen command line parameters for
> > > Xilinx boards: sched=null vwfi=native, then if the guest uses WFI,
> > > the CPU will execute WFI directly and go into low power mode.
> > Yes, using these command line params.
> >
> > > Given the issue you are describing, I am suspecting the guest is
> > > not issuing
> > > WFI: that is simple and known to work. Instead, I suspect that
> > > Linux might be trying to use PSCI_suspend in a way that is not
> > > supported or well- implemented by Xen.
> > >
> > > Can you check? You can add a printk in Linux
> > > drivers/firmware/psci/psci.c:__psci_cpu_suspend or in Xen
> > > xen/arch/arm/vpsci.c:do_psci_0_2_cpu_suspend
> > Instrumented both places it doesn't appear to reach there.  In
> > kernel/power/suspend.c, there's a call to s2idle_loop that it's currently 
> > 'stuck'
> > in and I think it doesn't get to the psci suspend your referring
> > till afterwards, when suspend_ops->enter is called.  Unfortunately,
> > without any idle-states nodes in the FDT, the only suspend state
> > Linux is defaults to is 'suspend to idle'.
>
> The fact that Linux uses "suspend to idle" is not a problem because as
> I mentioned WFI or PSCI_suspent are not different on Xen. That part is OK.
What if using "suspend to idle" is preventing a WFI/PSCI_suspend?  Which is
what I believe I'm currently seeing in my setup.  In kernel/power/suspend.c,
suspend_devices_and_enter(), it gets into the this s2idle_loop and upon
resuming from idle, it jumps past the point where I believe a
WFI/PSCI_suspend can happen.
if (state == PM_SUSPEND_TO_IDLE) {
                s2idle_loop();
                goto Platform_wake;
}

> However, if the issue is not PSCI_suspend then I don't have another
> easy guess. Please post a full stack trace or more information about
> the error in Linux and I might be able to see where it is coming from.

echo mem > /sys/power/state
[   75.666385] PM: suspend entry (s2idle)
[   75.685382] Filesystems sync: 0.018 seconds
[   75.685446] Preparing system for sleep (s2idle)
[   75.695087] Freezing user space processes ... (elapsed 0.001 seconds) done.
[   75.698339] OOM killer disabled.
[   75.701618] Freezing remaining freezable tasks ... (elapsed 0.063 seconds) 
done.
[   75.767215] Suspending system (s2idle)
[   75.777682] macb ff0e0000.ethernet: gem-ptp-timer ptp clock unregistered.
[   75.784525] suspend-to-idle
[   75.784580] CPU: 0 PID: 548 Comm: sh Not tainted 5.15.0-xilinx-v2022.1 #1
[   75.788626] Hardware name: xlnx,zynqmp (DT)

[   75.792867] Call trace:
[   75.795373]  dump_backtrace+0x0/0x1a8
[   75.799094]  show_stack+0x18/0x30
[   75.802466]  dump_stack_lvl+0x7c/0xa0
[   75.806186]  dump_stack+0x18/0x34
[   75.809559]  suspend_devices_and_enter+0x334/0x4cc
[   75.814403]  pm_suspend+0x1f0/0x248
[   75.817950]  state_store+0xb8/0x100
[   75.821497]  kobj_attr_store+0x1c/0x30
[   75.825302]  sysfs_kf_write+0x40/0x54
[   75.829022]  kernfs_fop_write_iter+0xb8/0x170
[   75.833433]  new_sync_write+0x74/0xc8
[   75.837153]  vfs_write+0x104/0x128
[   75.840613]  ksys_write+0x70/0xdc
[   75.843987]  __arm64_sys_write+0x18/0x24
[   75.847965]  invoke_syscall+0x74/0xfc
[   75.851684]  el0_svc_common.constprop.0+0xb0/0xdc
[   75.856442]  do_el0_svc+0x84/0x90
[   75.859816]  el0_svc+0x20/0x4c
[   75.862930]  el0t_64_sync_handler+0xbc/0x158
[   75.867254]  el0t_64_sync+0x1a0/0x1a4

CONFIDENTIALITY NOTICE: This e-mail, including any attachments, may contain 
information that is confidential and privileged. Any unauthorized disclosure, 
reproduction or use of this e-mail is prohibited. If you are not the intended 
recipient, please notify the sender by reply e-mail or telephone and 
permanently delete this e-mail and any reproductions immediately.