Re: Xen vs Trenchboot: TXT AP bringup

[Ross Philipson <ross.philipson@xxxxxxxxxx>]

On 11/3/22 05:29, 'Jan Beulich' via trenchboot-devel wrote:
> On 27.10.2022 21:37, Andrew Cooper wrote:
> > However, we're also very close to supporting parallel boot.  The
> > serialising point we currently have is __high_start loading %rsp from
> > stack_start, because that's a single pointer adjusted by do_boot_cpu().
> > Everything else, even the processor's idea of Xen's CPU id is derived
> > from the stack; specifically in the cpu_info block on the top of the
> > stack, which is prepared by the BSP.
> >
> > All we need to do full parallel boot is an array mapping APIC_ID -> Xen
> > CPU index.  The AP boot path doesn't use the trampoline stack at all.
> > Given some way of transforming APIC_ID into a Xen CPU index, the APs
> > could pull their stack pointer straight out of the stack_bases[] array.
> > This would allow us in the general case to boot APs using an
> > all-but-self INIT-SIPI-SIPI.  (This is how firmware does AP bringup anyway.)
> >
> >
> > Thoughts?  In particular, is there an obvious option I've missed[1]?
> Well, I don't see other options, but I'm not sure we're as close as you
> describe. I'm pretty sure we have assumptions elsewhere that multiple
> CPUs won't be starting in parallel. Those places will need finding and
> adjusting. Plus the generic CPU infrastructure also assumes things are
> done one by one, with (for example) cpu_up() taking a CPU number, not a
> CPU mask.
>
> Jan
For what it is worth, I can explain what I do for the Linux kernel. 
After SENTER transfers control to the setup kernel (the MLE), I wake all 
the APs up, restore their state, disable interrupts and park them (hlt) 
in a safe area that was setup.
Later in the SMP boot code I detect that it is a Secure Launch. This 
code is being called per CPU with the APIC IDs. One at a time I bypass 
the normal boot path (INIT-SIPI-SIPI or direct 64b) and call our SMP 
function. This function sends an IPI to the particular APIC in question 
(NMI in this case) and vectors it to the normal SMP startup code that is 
in the rmpiggy. From there everything else happens as it normally would 
and the AP jumps through the unchanged startup trampolines. There is 
actually not very much Secure Launch specific code in the SMP boot code. 
This is the one patch (well commit where the patch came from):
https://github.com/TrenchBoot/linux/commit/c81d3580bea8f0111e17ad0cdfce15149b46249f

And for anyone not familiar with the rmpiggy, it is a blob of machine 
code that is fixed up at runtime by the kernel proper and placed at an a 
location < 1M so INIT-SIPI-SIPI can vector to it in real mode.
Ross