Re: Xen on rk3399

[Bertrand Marquis <Bertrand.Marquis@xxxxxxx>]

Hi Brad

> On 12 Jul 2022, at 17:24, Brad Churchwell <brad@xxxxxxxxxxxx> wrote:
>
> Hi Bertrand,
>
> I believe I understand, but just to clarify, should I leave the 
> ppi-partitions block in rk3399.dtsi as is and disable the little cores, or 
> should I also modify that block?

Disable the little cores and remove the partitions.

Cheers
Bertrand

>
> Brad
>
> On Tue, Jul 12, 2022 at 11:11 AM Bertrand Marquis <Bertrand.Marquis@xxxxxxx> 
> wrote:
> Hi Brad,
>
> > On 12 Jul 2022, at 16:59, Brad Churchwell <brad@xxxxxxxxxxxx> wrote:
> >
> > Hi Bertrand,
> >
> > Thanks so much for the quick response!
> >
> > I should have mentioned previously that this device tree and kernel Image 
> > (5.15.16) does boot properly with the rootfs without XEN. The interrupt 
> > errors are only present when booting with XEN.
> >
> > These are custom boards and they do have usb c, however we are unable to 
> > boot from usb as it doesn't receive power.  We currently only use the usb 
> > to flash u-boot to spi which requires us to use a separate power adapter. 
> > These boards get power and networking through a custom backplane and 
> > interface with the backplane via a sata style justice (I'm not sure what 
> > the proper terminology would be here).
> >
> > Since I cannot boot to rootfs with XEN, I'm unable to show the device tree 
> > there. The only bit that I added to accommodate XEN is this bit here:
> >
> > chosen {
> >     stdout-path = "serial2:1500000n8";
> >     bootargs = "hmp-unsafe=true";
> >     xen,dom0-bootargs = "console=hvc0 earlyprintk=xen clk_ignore_unused 
> > root=/dev/nvme0n1p2 rw init=/sbin/init rootwait rootfstype=ext4 
> > ignore_loglevel";
> >     modules {
> >         #address-cells = <2>;
> >         #size-cells = <2>;
> >
> >         module@1 {
> >             compatible = "xen,linux-zimage", "xen,multiboot-module";
> >             reg = <0x0 0x03F80000 0x0 0x01aa8008>;
> >         };
> >         module@2 {
> >             compatible = "xen,linux-initrd", "xen,multiboot-module";
> >             reg = <0x0 0x06000000 0x0 0x02000000>;
> >         };
> >     };
> >
> > };
> >
> > The gic declaration is in the vanilla rk3399.dtsi file and hasn't been 
> > modified:
> >
> >         gic: interrupt-controller@fee00000 {
> >                 compatible = "arm,gic-v3";
> >                 #interrupt-cells = <4>;
> >                 #address-cells = <2>;
> >                 #size-cells = <2>;
> >                 ranges;
> >                 interrupt-controller;
> >
> >                 reg = <0x0 0xfee00000 0 0x10000>, /* GICD */
> >                       <0x0 0xfef00000 0 0xc0000>, /* GICR */
> >                       <0x0 0xfff00000 0 0x10000>, /* GICC */
> >                       <0x0 0xfff10000 0 0x10000>, /* GICH */
> >                       <0x0 0xfff20000 0 0x10000>; /* GICV */
> >                 interrupts = <GIC_PPI 9 IRQ_TYPE_LEVEL_HIGH 0>;
> >                 its: interrupt-controller@fee20000 {
> >                         compatible = "arm,gic-v3-its";
> >                         msi-controller;
> >                         #msi-cells = <1>;
> >                         reg = <0x0 0xfee20000 0x0 0x20000>;
> >                 };
> >
> >                 ppi-partitions {
> >                         ppi_cluster0: interrupt-partition-0 {
> >                                 affinity = <&cpu_l0 &cpu_l1 &cpu_l2 
> > &cpu_l3>;
> >                         };
> >
> >                         ppi_cluster1: interrupt-partition-1 {
> >                                 affinity = <&cpu_b0 &cpu_b1>;
> >                         };
> >                 };
>
> Xen is not supporting PPI partitions.
> I also saw that your booting using SMP unsafe so all core are enabled by Xen.
>
> Right now big/little is not supported by Xen and here you might have some 
> interrupts in your system which can only occur on some specific cores.
> I would suggest to try the following:
> - only boot enable in your dtb the cores with the same type than the boot core
> - check in your device tree which devices are assign to the ppi_cluster not 
> available and disable them
>
> This kind of configuration will be tricky to handle so it will depend on 
> which devices have interrupts that can only be fired on one cluster type.
>
> Sorry but I cannot right now check in the device tree and do those checks 
> myself but if you do get the idea send back a mail :-)
>
> Cheers
> Bertrand
>
> >         };
> >
> > I've attached the device tree and include files just in case you'd like to 
> > take a look at those.
> >
> > Thanks so much for your help! I've been at this for weeks with very little 
> > progress.
> >
> > Cheers,
> > Brad
> >
> >
> > On Tue, Jul 12, 2022 at 10:05 AM Bertrand Marquis 
> > <Bertrand.Marquis@xxxxxxx> wrote:
> > Hi Brad,
> >
> > > On 11 Jul 2022, at 19:38, Brad Churchwell <brad@xxxxxxxxxxxx> wrote:
> > >
> > > Hello,
> > >
> > > I've been trying to get Xen to boot dom0 with my kernel for weeks on an 
> > > rk3399 based board and thought I'd reach out for help. It looks like 
> > > either Xen is not properly recreating the device tree or the interrupt 
> > > controller is just failing. The hypervisor boots but falls to initramfs 
> > > because it cannot find the root device (nvme on pcie). Any help would be 
> > > greatly appreciated. Here is the complete boot log
> >
> > From the logs you have an issue with the interrupt controller and the fact 
> > that your NVME is behind PCIE and as such depends on ITS is probably not 
> > helping.
> >
> > I would suggest to try to boot on usb as root fs for a try.
> >
> > Also it could be useful to compare the device tree on xen and without xen 
> > to understand what is going on (using /proc/device-tree).
> >
> > Xen seems to be ok but Linux is not happy with interrupts and is showing 
> > several issues around this area on your logs.
> > Could you show us an extract of your device tree around the gic declaration 
> > ?
> >
> > Cheers
> > Bertrand
> >
> >
> > IMPORTANT NOTICE: The contents of this email and any attachments are 
> > confidential and may also be privileged. If you are not the intended 
> > recipient, please notify the sender immediately and do not disclose the 
> > contents to any other person, use it for any purpose, or store or copy the 
> > information in any medium. Thank you.
> > <rk3399-zaku.dts><rk3399-zaku.dtsi><rk3399.dtsi>
>
> IMPORTANT NOTICE: The contents of this email and any attachments are 
> confidential and may also be privileged. If you are not the intended 
> recipient, please notify the sender immediately and do not disclose the 
> contents to any other person, use it for any purpose, or store or copy the 
> information in any medium. Thank you.

IMPORTANT NOTICE: The contents of this email and any attachments are 
confidential and may also be privileged. If you are not the intended recipient, 
please notify the sender immediately and do not disclose the contents to any 
other person, use it for any purpose, or store or copy the information in any 
medium. Thank you.