Re: Xen PVH domU start-of-day VCPU state

[Martin Lucina <martin@xxxxxxxxxx>]

Oh! I think I've found a solution, even though I don't entirely understand the
problem/root cause:

On Tuesday, 26.05.2020 at 12:12, Martin Lucina wrote:
> > On Tue, May 26, 2020 at 11:34:21AM +0200, Roger Pau Monné wrote:
> > Forgot to ask, but can you also add the output of readelf -lW
> > <kernel>?
> 
>     Elf file type is EXEC (Executable file)
>     Entry point 0x1001e0
>     There are 7 program headers, starting at offset 64
> 
>     Program Headers:
>       Type           Offset   VirtAddr           PhysAddr           FileSiz  
> MemSiz   Flg Align
>       INTERP         0x001000 0x0000000000100000 0x0000000000100000 0x000018 
> 0x000018 R   0x8
>           [Requesting program interpreter: /nonexistent/solo5/]
>       LOAD           0x001000 0x0000000000100000 0x0000000000100000 0x00626c 
> 0x00626c R E 0x1000
>       LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x007120 
> 0x00ed48 RW  0x1000
>       NOTE           0x0080ac 0x00000000001070ac 0x00000000001070ac 0x000018 
> 0x000018 R   0x4
>       NOTE           0x00f120 0x00000000001070c4 0x00000000001070c4 0x000014 
> 0x000000 R   0x4
                                                                               
^^^^^^^^

I should have picked up on the above, but thought it wasn't relevant.

>       NOTE           0x008088 0x0000000000107088 0x0000000000107088 0x000024 
> 0x000024 R   0x4
>       NOTE           0x008000 0x0000000000107000 0x0000000000107000 0x000088 
> 0x000088 R   0x4
> 
>      Section to Segment mapping:
>       Segment Sections...
>        00     .interp
>        01     .interp .text .rodata .eh_frame
>        02     .note.solo5.manifest .note.solo5.abi .note.solo5.not-openbsd 
> .data .bss

And also the missing .note.solo5.xen above.

>        03     .note.solo5.not-openbsd
>        04     .note.solo5.xen
>        05     .note.solo5.abi
>        06     .note.solo5.manifest

Turns out that the .note.solo5.xen section as defined in boot.S was not
marked allocatable, and that was doing <something> that was confusing our
linker script[1] (?).

If I make this simple change:

--- a/bindings/xen/boot.S
+++ b/bindings/xen/boot.S
@@ -32,7 +32,7 @@
 #define ENTRY(x) .text; .globl x; .type x,%function; x:
 #define END(x)   .size x, . - x

-.section .note.solo5.xen
+.section .note.solo5.xen, "a", @note

        .align  4
        .long   4

then I get the expected output from readelf -lW, and I can get as far as
the C _start() with no issues!

FWIW, here's the diff of readelf -lW before/after:

--- before      2020-05-26 17:36:46.117885855 +0200
+++ after       2020-05-26 17:38:07.090508322 +0200
@@ -8,9 +8,9 @@
   INTERP         0x001000 0x0000000000100000 0x0000000000100000 0x000018 
0x000018 R   0x8
       [Requesting program interpreter: /nonexistent/solo5/]
   LOAD           0x001000 0x0000000000100000 0x0000000000100000 0x00615c 
0x00615c R E 0x1000
-  LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x007120 
0x00ed28 RW  0x1000
+  LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x006120 
0x00dd28 RW  0x1000
   NOTE           0x0080ac 0x00000000001070ac 0x00000000001070ac 0x000018 
0x000018 R   0x4
-  NOTE           0x00f120 0x00000000001070c4 0x00000000001070c4 0x000014 
0x000000 R   0x4
+  NOTE           0x0080c4 0x00000000001070c4 0x00000000001070c4 0x000014 
0x000014 R   0x4
   NOTE           0x008088 0x0000000000107088 0x0000000000107088 0x000024 
0x000024 R   0x4
   NOTE           0x008000 0x0000000000107000 0x0000000000107000 0x000088 
0x000088 R   0x4

@@ -18,7 +18,7 @@
   Segment Sections...
    00     .interp
    01     .interp .text .rodata .eh_frame
-   02     .note.solo5.manifest .note.solo5.abi .note.solo5.not-openbsd .data 
.bss
+   02     .note.solo5.manifest .note.solo5.abi .note.solo5.not-openbsd 
.note.solo5.xen .data .bss
    03     .note.solo5.not-openbsd
    04     .note.solo5.xen
    05     .note.solo5.abi

-mato

[1] https://github.com/mato/solo5/blob/xen/bindings/xen/solo5_xen.lds