Xen project Mailing List

Hey Julien,

Would you mind sharing the latest version of your code?

Of course not. This is the current version:

asmlinkage void leave_hypervisor_tail(void)
{
+    /*This methode will be called after the 'guest_entry' macro in /arch/arm64/entry.S set guest registers
+    Check single_step_enabled flag in domain struct here and set needed registers
+    */
+
+    struct vcpu *v = current;
+
+    if ( unlikely(v->domain->arch.monitor.singlestep_enabled ) )
+    {
+
+
+
+        if(!(guest_cpu_user_regs()->cpsr & 0b1000))
+        {
+            WRITE_SYSREG(READ_SYSREG(MDSCR_EL1) | 0x1, MDSCR_EL1);
+            WRITE_SYSREG(READ_SYSREG(MDCR_EL2) | HDCR_TDE, MDCR_EL2);
+            guest_cpu_user_regs()->cpsr = guest_cpu_user_regs()->cpsr | 0x200000;
+            WRITE_SYSREG( READ_SYSREG(DAIF) & ~0x200, DAIF);
+            isb();
+            v->arch.single_step = 1;
+
+        }
+
+
+    }else
+    {
+        //single_step Domain flag not set
+        if( v->arch.single_step )
+        {
+            gprintk(XENLOG_ERR, "Domain flag not set, but vcpu flag is set\n");
+            WRITE_SYSREG(READ_SYSREG(MDSCR_EL1) & ~0x1, MDSCR_EL1);
+            guest_cpu_user_regs()->cpsr = guest_cpu_user_regs()->cpsr & ~0x200000;
+            //WRITE_SYSREG(READ_SYSREG(SPSR_EL2) | 0x200000, SPSR_EL2 );
+            WRITE_SYSREG( READ_SYSREG(DAIF) & ~0x200, DAIF);
+            v->arch.single_step = 0;
+        }
+
+    }
+
     while (1)
     {
         local_irq_disable();

This code does still the same. Check if domain flag is set, if so check if SS bit has to be set. If Domain flag is not set, clear SS related bits.

Did you look where the PCs point to? Is it your kernel module?

Yes. I compared it with my Linux Image (using objdump) and found that these instructions are within a spinlock (the function is called _raw_spin_lock to be exact). My module is always loaded around the address 0xffff0000008e0000. Also I could see that its not the case that the addresses are printed twice but the VM just keeps within the spinlock (which created problems with printing).

By tracing every single step I could determine the function sequence that is called. It starts out with an <el1_irq> and stops with an <hrtimer_interrupt> that keeps getting locked in <_raw_spin_lock>.

In order to solve this, I compared my solution with the KVM one, where I saw (at least for my understanding) that they disable Interrupts for the VM. In the KVM file: /kvm/virt/arm/arm.c the function "kvm_arch_vcpu_ioctl_run" handles the running of the VM. The "kvm_arm_setup_debug" function does the same steps as I do in order to enable software step exceptions.

So I can't see any difference there.

I also modified xen-access so that the singlestepp will be startet with an SMC from the Guest. Additionally i wrote a second test kernel module which only executes an SMC and than will be stopped. I can trace the two SMCs and than the course to the spinlock (I put the trace below). While in the spinlock the VM won't response to anything. But after disabling singlestep it starts to work again.

Enabled singlestep directly (not with an SMC) results in the VM to be locked in the spinlock immediately.

There is also a problem with using printk within my module for the same reason. It will always end in the spinlock.

One reason I could imaging is that because I'm singlestepping everything, including timer interrupts, there will be problems with the scheduling of the VM. This results in, not meeting the conditions to exit the spinlock.

I hope i made the situation as clear as possible.

Thank you for your help

Florian

This is the function trace obtained by singlestepping every instruction until the system reached the spinlock below. The two SMCs from my Module are needed for my xen-access implementation to ensure that the SS starts withing my module

root@avocet:~# ./xen-access -m 1 singlestep
Singlestep request found
xenaccess init
max_gpfn = 60000
Run singlestep with commands? (y/n):y
Starting loop
Privileged call: pc=ffff0000008e0000 (vcpu 0)
Privileged call: pc=ffff0000008e0004 (vcpu 0)

Singlestep: PC=ffff000008081a80     b    ffff000008082700 <el1_irq>
Singlestep: PC=ffff000008082700     <el1_irq>
[...]
Singlestep: PC=ffff0000080814e8        <gic_handle_irq>:
[...]
Singlestep: PC=ffff000008100f60     <__handle_domain_irq>:
[...]
Singlestep: PC=ffff0000080c1708     <irq_enter>:
[...]
Singlestep: PC=ffff00000810fbf8        <rcu_irq_enter>:
[...]
Singlestep: PC=ffff0000080c1714     return to <irq_enter>
[...]
Singlestep: PC=ffff000008100f98     return to <__handle_domain_irq>:
[...]
Singlestep: PC=ffff000008108238     <irq_find_mapping>:
[...]
Singlestep: PC=ffff000008100ff4     return to <__handle_domain_irq>:
[...]
Singlestep: PC=ffff000008100928     <generic_handle_irq>:
[...]
Singlestep: PC=ffff00000837d0f8     <radix_tree_lookup>:
[...]
Singlestep: PC=ffff00000837cfd8     return to <generic_handle_irq>:
[...]
Singlestep: PC=ffff00000837d000     return to <radix_tree_lookup>:
[...]
Singlestep: PC=ffff000008100940     return to <generic_handle_irq>:
[...]
Singlestep: PC=ffff000008105b18     handle_percpu_devid_irq
[...]
Singlestep: PC=ffff0000087788c8     <arch_timer_handler_virt>: /linux/driver/clocksource/arm_arch_timer.c
[...]
Singlestep: PC=ffff000008115108     <hrtimer_interrupt>:
[...]
Singlestep: PC=ffff0000088cbd50     <_raw_spin_lock>:
Singlestep: PC=ffff0000088cbd54
Singlestep: PC=ffff0000088cbd58
Singlestep: PC=ffff0000088cbd5c
Singlestep: PC=ffff0000088cbd60

Singlestep: PC=ffff0000088cbd64
Singlestep: PC=ffff0000088cbd68        This code block gets repeated until SS disabled
Singlestep: PC=ffff0000088cbd6c
Singlestep: PC=ffff0000088cbd70

Singlestep: PC=ffff0000088cbd64
Singlestep: PC=ffff0000088cbd68
Singlestep: PC=ffff0000088cbd6c
Singlestep: PC=ffff0000088cbd70

Re: [Xen-devel] xen/arm: Software Step ARMv8 - PC stuck on instruction