Re: [Xen-devel] questions of vm save/restore on arm64

[Chenxiao Zhao <chenxiao.zhao@xxxxxxxxx>]

On 6/12/2016 11:31 PM, Julien Grall wrote:
> On 12/06/2016 10:46, Chenxiao Zhao wrote:
> > Hi all,
> Hello,
>
> > I finally got save/restore working on arm64, but it only works when I
> > assign only one vCPU to VM. If I set vcpus=4 in configure file, the
> > restored VM does not work properly.
> Can you describe what you mean by "does not work properly"? What are the
> symptoms?
After restoring VM with more than one vCPU, the VM keeps in "b" state.

I'm running Centos on guest and listed the console log after restore.

[   32.530490] Xen: initializing cpu0
[   32.530490] xen:grant_table: Grant tables using version 1 layout
[   32.531034] PM: noirq restore of devices complete after 0.382 msecs
[   32.531382] PM: early restore of devices complete after 0.300 msecs
[   32.531430] Xen: initializing cpu1
[   32.569028] PM: restore of devices complete after 24.663 msecs
[   32.569304] Restarting tasks ...
[   32.569903] systemd-journal[800]: undefined instruction: 
pc=0000007fa37dd4c8
[   32.569975] Code: 2947b0cb d50339bf b94038c9 d5033fdf (d53be04f)
[   32.571530] done.
[   32.571631] systemd[1]: undefined instruction: pc=0000007f8a9ea4c8
[   32.571650] Code: 2947b0cb d50339bf b94038c9 d5033fdf (d53be04f)
[   32.573527] auditd[1365]: undefined instruction: pc=0000007f8aca24c8
[   32.573553] Code: 2947b0cb d50339bf b94038c9 d5033fdf (d53be04f)
[   32.636573] systemd-cgroups[2210]: undefined instruction: 
pc=0000007f99ad14c8
[   32.636633] Code: 2947b0cb d50339bf b94038c9 d5033fdf (d53be04f)
[   32.636726] audit: *NO* daemon at audit_pid=1365
[   32.636741] audit: audit_lost=1 audit_rate_limit=0 
audit_backlog_limit=320
[   32.636755] audit: auditd disappeared
[   32.638545] systemd-logind[1387]: undefined instruction: 
pc=0000007f86e5b4c8
[   32.638594] Code: 2947b0cb d50339bf b94038c9 d5033fdf (d53be04f)
[   32.638673] audit: type=1701 audit(68.167:214): auid=4294967295 uid=0 
gid=0 s
es=4294967295 subj=system_u:system_r:systemd_logind_t:s0 pid=1387 
comm="systemd-
logind" exe="/usr/lib/systemd/systemd-logind" sig=4
[   32.647972] systemd-cgroups[2211]: undefined instruction: 
pc=0000007fa7f414c8
[   32.648017] Code: 2947b0cb d50339bf b94038c9 d5033fdf (d53be04f)
[   32.648087] audit: type=1701 audit(68.177:215): auid=4294967295 uid=0 
gid=0 s
es=4294967295 subj=system_u:system_r:init_t:s0 pid=2211 
comm="systemd-cgroups" e
xe="/usr/lib/systemd/systemd-cgroups-agent" sig=4
[   61.401838] do_undefinstr: 8 callbacks suppressed
[   61.401882] crond[1550]: undefined instruction: pc=0000007f8d15d4c8
[   61.401903] Code: 2947b0cb d50339bf b94038c9 d5033fdf (d53be04f)
[   61.402077] audit: type=1701 audit(96.947:218): auid=4294967295 uid=0 
gid=0 s
es=4294967295 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 pid=1550 
comm="crond
" exe="/usr/sbin/crond" sig=4
[   61.407024] dbus-daemon[1390]: undefined instruction: pc=0000007f87fae4c8
[   61.407064] Code: 2947b0cb d50339bf b94038c9 d5033fdf (d53be04f)
[   61.407212] audit: type=1701 audit(96.947:219): auid=4294967295 
uid=81 gid=81
 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 
pid=1390 co
mm="dbus-daemon" exe="/usr/bin/dbus-daemon" sig=4
[   61.408311] systemd-cgroups-agent[2214]: Failed to process message 
[type=erro
r sender=org.freedesktop.DBus path=n/a interface=n/a member=n/a 
signature=s]: Co
nnection timed out
[   61.416815] systemd-cgroups-agent[2216]: Failed to get D-Bus 
connection: Conn
ection refused
[   61.421499] systemd-cgroups-agent[2215]: Failed to get D-Bus 
connection: Conn
ection refused
[   61.429413] systemd-cgroups-agent[2217]: Failed to get D-Bus 
connection: Conn
ection refused
[   61.434301] systemd-cgroups-agent[2218]: Failed to get D-Bus 
connection: Conn
ection refused
[   61.435016] systemd-cgroups-agent[2219]: Failed to get D-Bus 
connection: Conn
ection refused

[  110.095570] audit: type=1701 audit(145.637:220): auid=0 uid=0 gid=0 
ses=1 sub
j=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=2189 
comm="bash" exe
="/usr/bin/bash" sig=4
[  110.098120] audit: type=1104 audit(145.637:221): pid=1602 uid=0 
auid=0 ses=1
subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=PAM:setcred 
grantors
=pam_securetty,pam_unix acct="root" exe="/usr/bin/login" hostname=? 
addr=? termi
nal=hvc0 res=success'
[  110.102730] audit: type=1106 audit(145.637:222): pid=1602 uid=0 
auid=0 ses=1
subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 
msg='op=PAM:session_close gr
antors=? acct="root" exe="/usr/bin/login" hostname=? addr=? 
terminal=hvc0 res=fa
iled'
[  110.112341] systemd-cgroups-agent[2220]: Failed to get D-Bus 
connection: Conn
ection refused

> Also, I would start by debugging with 2 vCPUs and then increasing the
> number step by step.
It's the same issue when restoring VM with more than one vCPUS. What I 
see is guest reported "undefined instruction" with random PC depends on 
the save point.
Can you advice how would I start debugging this issue?

Thanks.

> Regards,
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel