Xen project Mailing List

Hello James, Julien,

regarding your previous mails. I was able to single step every instruction of my module. The problem (or rather the solution) was to _disable_ the IRQ interrupts from within my guest module. This solves the problem of singlestepping a module which previously ended in a spinlock. But it does not solve the problem with system that is singlestepped with enabled IRQ's, as it still will be locked within a spinlock.

My module starts the singlestep routine by executing an SMC instruction. This will be catched from within my xen-access in the Dom0 (which also will increase the PC accordingly). Xen-access will then trigger an event to set the domain flag for my VM (the Domain flag determines if SS should be enabled when entering the guest) and by this start singlestepping. Every SS exception is also caught within xen-access which then lets me decide how to continue the execution.

This is my module code which is executed in the DomU:

int init_module()
{
    printk(KERN_INFO "###     Init address 0x%lx\n", &init_module);
    printk(KERN_INFO "        Set function hook\n");
    patch_function_hook();

    printk(KERN_INFO "        Starting singlestep\n");

This following part is the importand bit. I disable the local_irq's and execute two SMCs.

    local_irq_disable();
    __asm__ __volatile__ ("SMC 1");
    __asm__ __volatile__ ("SMC 1");

    if(!already_trapped)
    {
      __asm__ __volatile__ ("NOP");
        __asm__ __volatile__ ("NOP");
        __asm__ __volatile__ ("NOP");

        //Just for keeping module busy while singlestep
        for ( c = 0 ; c < n ; c++ )

When executing these exact steps, it is possible to singlestep the whole module. Without the local_irq_disable() the system will stop the module execution right after the first SMC.

My Xen Access version got, just like the original one, a main loop where events will be catched:

    printf("Starting loop\n" );
    for (;;)
    {
        if ( interrupted )
        {
            [...]
        }

        while ( RING_HAS_UNCONSUMED_REQUESTS(&xenaccess->vm_event.back_ring) )
        {
          [...]

            switch (req.reason) {
            case VM_EVENT_REASON_PRIVILEGED_CALL:
                printf("++++++++++++++++++++++++++++Privileged call: pc=%"PRIx64" (vcpu %d)\n",
                       req.data.regs.arm.pc,
                       req.vcpu_id);

                if (!ss_started)
                {
                    ss_started = 1;
                    toggle_single_step_domain(1);

This part enables singlestepping the VM with the first SMC. (The function toggle_single_step_domain just sets the domain flag i mentioned in earlier mails)

                }else
                {
                    printf("Signlestep already activated\n");
                }

                rsp.data.regs.arm = req.data.regs.arm;
                rsp.data.regs.arm.pc += 4;

This is the part which sets the VM.PC to the next instruction after the SMC instruction within the guest module.

            case VM_EVENT_REASON_SINGLESTEP:

                printf("Singlestep: PC=%016"PRIx64", vcpu %d, \n", req.data.regs.arm.pc,req.vcpu_id);


                if(!run)
                {

                    printf("Enter command: s:step; c:end; r:regs:;x:end :" );

                    do
                    {
                        command = getchar();
                    }
                    while (isspace(command));

                    switch(command)
                    {
                        case 'c':   printf("Singlestep weiterlaufen \n");
                                   // rsp.flags |= VM_EVENT_FLAG_TOGGLE_SINGLESTEP;
                                    run = 1;
                                    break;
                          [More possible commands here]
                        default:    printf("Wrong command \"%c\"!\n", command );
                                    break;

                    }

The above part is the handling of software step exceptions. As you can see, i print the PC value and read a command from the user in order to let the guest execute further or print its registers.

These two are the main control flow of xen-access.

Finally my Xen Code within the leave_hypervisor_tail looks like this. The leave_hypervisor_tail function is called right before the control flow switches to the guest. Because of this, every time when the guest gets to run, we set the bits needed to singlestep.

asmlinkage void leave_hypervisor_tail(void)
{
    struct vcpu *v = current;

    if ( unlikely(v->domain->arch.monitor.singlestep_enabled ) )
    {

        if(!(guest_cpu_user_regs()->cpsr & 0b1000))
        {

            WRITE_SYSREG(READ_SYSREG(MDSCR_EL1) | 0x1, MDSCR_EL1);
            WRITE_SYSREG(READ_SYSREG(MDCR_EL2) | HDCR_TDE, MDCR_EL2);
            guest_cpu_user_regs()->cpsr = guest_cpu_user_regs()->cpsr | 0x200000;

            v->arch.single_step = 1;

        }


    }else
    {
        //single_step Domain flag not set
        if( v->arch.single_step )
        {
            gprintk(XENLOG_ERR, "Domain flag not set, but vcpu flag is set\n");
            WRITE_SYSREG(READ_SYSREG(MDSCR_EL1) & ~0x1, MDSCR_EL1);
            guest_cpu_user_regs()->cpsr = guest_cpu_user_regs()->cpsr & ~0x200000;

            v->arch.single_step = 0;
        }

    }

    while (1)
    {
        local_irq_disable();

[...]

So, I adapted your comments and as I mentioned. The general SingleStep functionality works fine.

But: It's not possible to singlestep a system as long as the VM IRQ's are enabled. If we would activate single stepping with enabled interrupts, we will be locked in the mentioned spinlock.

Because of this it is not possible to singlestep other application. Additionally it is not possible to print anything while singlestepping because, as far as I understood, the system will wait within a spinlock until the terminal is free to print.

Do you have any idea why it's not possible to escape the lock while singlestepping? Like I mentioned, my guess is on timer interrupts, which should unlock the spinlock but generate problems with singlestep enabled at the same time. This would also explain why i can observe the control flow of my guest module with IRQ's being disabled.

Below is an abstract of the program flow. The first one is with disabled IRQ's. You can see that every instruction is from my module (as the addresses are around 0xffff0000008e0000). The second one shows instructions which, directly after the SMC, starts the "go to spinlock" routine because of enabled IRQ's.

Kind regards,

Florian

Working Singlestep in module:

root@avocet:~# ./xen-access -m 2 singlestep
Singlestep request found
Starting loop
++++++++++++++++++++++++++++Privileged call: pc=ffff0000008e0618 (vcpu 0)
Seeting Singlestep to 1
SingleStep succesfull rc=0
++++++++++++++++++++++++++++Privileged call: pc=ffff0000008e061c (vcpu 0)
Singlestep: PC=ffff0000008e0624, vcpu 0,
Singlestep: PC=ffff0000008e0628, vcpu 0,
Singlestep: PC=ffff0000008e062c, vcpu 0,
Singlestep: PC=ffff0000008e0630, vcpu 0,
Singlestep: PC=ffff0000008e0634, vcpu 0,

Singlestep with enabled IRQ's:

root@avocet:~# ./xen-access -m 2 singlestep
Singlestep request found
Starting loop

++++++++++++++++++++++++++++Privileged call: pc=ffff0000008e0000 (vcpu 0)

(vcpu 0)
Seeting Singlestep to 1
SingleStep succesfull rc=0

Singlestep: PC=ffff000008081a80

[...]

Singlestep: PC=ffff000008082700
[...]

Singlestep: PC=ffff0000080814e8
[...]

Singlestep: PC=ffff000008100f60

Re: [Xen-devel] xen/arm: Software Step ARMv8 - PC stuck on instruction