Xen project Mailing List

Hi Bertrand,

I did this yesterday and it does indeed silence the warnings saying to disable those cores, however the IRQ errors still remain.

I think I made some progress by changing "interrupts = <GIC_PPI 9 IRQ_TYPE_LEVEL_HIGH 0>;" to "interrupts = <GIC_SPI 9 IRQ_TYPE_LEVEL_HIGH 0>;". I noticed when trying to use XEN-4.10 that it said "IRQ only through SPI" and failed to boot XEN where 4.17 unstable boots without this error. Now that I changed GIC_PPI to GIC_SPI, I get the following output:

zaku => run boot_pci_enum; run nvme_init; fatload nvme 0:1 0x02000000 xen4.10-uImage; fatload nvme 0:1 0x01f00000 rk3399-zaku.dtb; fatload nvme 0:1 0x03F80000 Image;fatload nvme 0:1 0
885136 bytes read in 5 ms (168.8 MiB/s)
56185 bytes read in 3 ms (17.9 MiB/s)
27885576 bytes read in 65 ms (409.1 MiB/s)
8674182 bytes read in 22 ms (376 MiB/s)
## Booting kernel from Legacy Image at 02000000 ...
Image Name:
Image Type: AArch64 Linux Kernel Image (uncompressed)
Data Size: 885072 Bytes = 864.3 KiB
Load Address: 02000000
Entry Point: 02000000
Verifying Checksum ... OK
## Flattened Device Tree blob at 01f00000
Booting using the fdt blob at 0x1f00000
Loading Kernel Image
Loading Device Tree to 00000000f1efd000, end 00000000f1f0db78 ... OK

Starting kernel ...

- UART enabled -
- CPU 00000000 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Zero BSS -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) Checking for initrd in /chosen
(XEN) RAM: 0000000000200000 - 00000000f7ffffff
(XEN)
(XEN) MODULE[0]: 00000000f1efd000 - 00000000f1f0b000 Device Tree
(XEN) MODULE[1]: 0000000003f80000 - 0000000005a28008 Kernel
(XEN) MODULE[2]: 0000000006000000 - 0000000008000000 Ramdisk
(XEN)
(XEN) Command line: <NULL>
(XEN) Placing Xen at 0x00000000f7e00000-0x00000000f8000000
(XEN) Update BOOTMOD_XEN from 0000000002000000-0000000002120d81 => 00000000f7e00000-00000000f7f20d81
(XEN) Domain heap initialised
(XEN) Booting using Device Tree
(XEN) Platform: Generic System
(XEN) Taking dtuart configuration from /chosen/stdout-path
(XEN) Looking for dtuart at "serial2", options "1500000n8"
Xen 4.10.1
(XEN) Xen version 4.10.1 (root@) (aarch64-linux-gnu-gcc (Linaro GCC 7.3-2018.05) 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701]) debug=y Tue J2
(XEN) Latest ChangeSet:
(XEN) Processor: 410fd034: "ARM Limited", variant: 0x0, part 0xd03, rev 0x4
(XEN) 64-bit Execution:
(XEN) Processor Features: 0000000001002222 0000000000000000
(XEN) Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
(XEN) Extensions: FloatingPoint AdvancedSIMD GICv3-SysReg
(XEN) Debug Features: 0000000010305106 0000000000000000
(XEN) Auxiliary Features: 0000000000000000 0000000000000000
(XEN) Memory Model Features: 0000000000001122 0000000000000000
(XEN) ISA Features: 0000000000011120 0000000000000000
(XEN) 32-bit Execution:
(XEN) Processor Features: 00000131:10011011
(XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle
(XEN) Extensions: GenericTimer Security
(XEN) Debug Features: 03010066
(XEN) Auxiliary Features: 00000000
(XEN) Memory Model Features: 10201105 40000000 01260000 02102211
(XEN) ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121
(XEN) Using SMC Calling Convention v1.0
(XEN) Using PSCI v1.0
(XEN) SMP: Allowing 4 CPUs
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 24000 KHz
(XEN) GICv3 initialization:
(XEN) gic_dist_addr=0x000000fee00000
(XEN) gic_maintenance_irq=41
(XEN) gic_rdist_stride=0
(XEN) gic_rdist_regions=1
(XEN) redistributor regions:
(XEN) - region 0: 0x000000fef00000 - 0x000000fefc0000
(XEN) GICv3 compatible with GICv2 cbase 0x000000fff00000 vbase 0x000000fff20000
(XEN) GICv3: 288 lines, (IID 0001143b).
(XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Allocated console ring of 32 KiB.
(XEN) Bringing up CPU1
- CPU 00000001 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) GICv3: CPU1: Found redistributor in region 0 @000000004003c000
(XEN) CPU 1 booted.
(XEN) Bringing up CPU2
- CPU 00000002 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) GICv3: CPU2: Found redistributor in region 0 @000000004005c000
(XEN) CPU 2 booted.
(XEN) Bringing up CPU3
- CPU 00000003 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) GICv3: CPU3: Found redistributor in region 0 @000000004007c000
(XEN) CPU 3 booted.
(XEN) Brought up 4 CPUs
(XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit VMID
(XEN) P2M: 3 levels with order-1 root, VTCR 0x80023558
(XEN) I/O virtualisation disabled
(XEN) build-id: b7279757e82e3e9ea4c4115dabdbfe25d56d762b
(XEN) alternatives: Patching with alt table 000000004018fe70 -> 0000000040190350
(XEN) grant_table.c:1731:IDLEv0 Expanding d0 grant table from 0 to 1 frames
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading kernel from boot module @ 0000000003f80000
(XEN) Loading ramdisk from boot module @ 0000000006000000
(XEN) Allocating 1:1 mappings totalling 512MB for dom0:
(XEN) BANK[0] 0x00000020000000-0x00000040000000 (512MB)
(XEN) Grant table range: 0x000000f7e00000-0x000000f7e40000
(XEN) IRQ 41 is already used by Xen
(XEN) Unable to map IRQ41 to dom0
(XEN) Device tree generation failed (-16).
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Could not set up DOM0 guest OS
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...

It looks like XEN is now using the interrupt controller but there are conflicts. I was able to get this error to change to "Unable to map IRQ43 to dom0" by changing all instances of "IRQ_TYPE_LEVEL_HIGH " to "IRQ_TYPE_LEVEL_LOW", but I can't get any further. I made sure that there was nothing in the device tree that is obviously using IRQ 43 as well.

Any thoughts on this?

Cheers,

Brad

On Wed, Jul 13, 2022, 8:00 AM Bertrand Marquis <Bertrand.Marquis@xxxxxxx> wrote:

Hi Brad

> On 12 Jul 2022, at 17:24, Brad Churchwell <brad@xxxxxxxxxxxx> wrote:
>
> Hi Bertrand,
>
> I believe I understand, but just to clarify, should I leave the ppi-partitions block in rk3399.dtsi as is and disable the little cores, or should I also modify that block?

Disable the little cores and remove the partitions.

Cheers
Bertrand

>
> Brad
>
> On Tue, Jul 12, 2022 at 11:11 AM Bertrand Marquis <Bertrand.Marquis@xxxxxxx> wrote:
> Hi Brad,
>
> > On 12 Jul 2022, at 16:59, Brad Churchwell <brad@xxxxxxxxxxxx> wrote:
> >
> > Hi Bertrand,
> >
> > Thanks so much for the quick response!
> >
> > I should have mentioned previously that this device tree and kernel Image (5.15.16) does boot properly with the rootfs without XEN. The interrupt errors are only present when booting with XEN.
> >
> > These are custom boards and they do have usb c, however we are unable to boot from usb as it doesn't receive power. We currently only use the usb to flash u-boot to spi which requires us to use a separate power adapter. These boards get power and networking through a custom backplane and interface with the backplane via a sata style justice (I'm not sure what the proper terminology would be here).
> >
> > Since I cannot boot to rootfs with XEN, I'm unable to show the device tree there. The only bit that I added to accommodate XEN is this bit here:
> >
> > chosen {
> > stdout-path = "serial2:1500000n8";
> > bootargs = "hmp-unsafe=true";
> > xen,dom0-bootargs = "console=hvc0 earlyprintk=xen clk_ignore_unused root=/dev/nvme0n1p2 rw init=/sbin/init rootwait rootfstype=ext4 ignore_loglevel";
> > modules {
> > #address-cells = <2>;
> > #size-cells = <2>;
> >
> > module@1 {
> > compatible = "xen,linux-zimage", "xen,multiboot-module";
> > reg = <0x0 0x03F80000 0x0 0x01aa8008>;
> > };
> > module@2 {
> > compatible = "xen,linux-initrd", "xen,multiboot-module";
> > reg = <0x0 0x06000000 0x0 0x02000000>;
> > };
> > };
> >
> > };
> >
> > The gic declaration is in the vanilla rk3399.dtsi file and hasn't been modified:
> >
> > gic: interrupt-controller@fee00000 {
> > compatible = "arm,gic-v3";
> > #interrupt-cells = <4>;
> > #address-cells = <2>;
> > #size-cells = <2>;
> > ranges;
> > interrupt-controller;
> >
> > reg = <0x0 0xfee00000 0 0x10000>, /* GICD */
> > <0x0 0xfef00000 0 0xc0000>, /* GICR */
> > <0x0 0xfff00000 0 0x10000>, /* GICC */
> > <0x0 0xfff10000 0 0x10000>, /* GICH */
> > <0x0 0xfff20000 0 0x10000>; /* GICV */
> > interrupts = <GIC_PPI 9 IRQ_TYPE_LEVEL_HIGH 0>;
> > its: interrupt-controller@fee20000 {
> > compatible = "arm,gic-v3-its";
> > msi-controller;
> > #msi-cells = <1>;
> > reg = <0x0 0xfee20000 0x0 0x20000>;
> > };
> >
> > ppi-partitions {
> > ppi_cluster0: interrupt-partition-0 {
> > affinity = <&cpu_l0 &cpu_l1 &cpu_l2 &cpu_l3>;
> > };
> >
> > ppi_cluster1: interrupt-partition-1 {
> > affinity = <&cpu_b0 &cpu_b1>;
> > };
> > };
>
> Xen is not supporting PPI partitions.
> I also saw that your booting using SMP unsafe so all core are enabled by Xen.
>
> Right now big/little is not supported by Xen and here you might have some interrupts in your system which can only occur on some specific cores.
> I would suggest to try the following:
> - only boot enable in your dtb the cores with the same type than the boot core
> - check in your device tree which devices are assign to the ppi_cluster not available and disable them
>
> This kind of configuration will be tricky to handle so it will depend on which devices have interrupts that can only be fired on one cluster type.
>
> Sorry but I cannot right now check in the device tree and do those checks myself but if you do get the idea send back a mail :-)
>
> Cheers
> Bertrand
>
> > };
> >
> > I've attached the device tree and include files just in case you'd like to take a look at those.
> >
> > Thanks so much for your help! I've been at this for weeks with very little progress.
> >
> > Cheers,
> > Brad
> >
> >
> > On Tue, Jul 12, 2022 at 10:05 AM Bertrand Marquis <Bertrand.Marquis@xxxxxxx> wrote:
> > Hi Brad,
> >
> > > On 11 Jul 2022, at 19:38, Brad Churchwell <brad@xxxxxxxxxxxx> wrote:
> > >
> > > Hello,
> > >
> > > I've been trying to get Xen to boot dom0 with my kernel for weeks on an rk3399 based board and thought I'd reach out for help. It looks like either Xen is not properly recreating the device tree or the interrupt controller is just failing. The hypervisor boots but falls to initramfs because it cannot find the root device (nvme on pcie). Any help would be greatly appreciated. Here is the complete boot log
> >
> > From the logs you have an issue with the interrupt controller and the fact that your NVME is behind PCIE and as such depends on ITS is probably not helping.
> >
> > I would suggest to try to boot on usb as root fs for a try.
> >
> > Also it could be useful to compare the device tree on xen and without xen to understand what is going on (using /proc/device-tree).
> >
> > Xen seems to be ok but Linux is not happy with interrupts and is showing several issues around this area on your logs.
> > Could you show us an extract of your device tree around the gic declaration ?
> >
> > Cheers
> > Bertrand
> >
> >
> > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
> > <rk3399-zaku.dts><rk3399-zaku.dtsi><rk3399.dtsi>
>
> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: Xen on rk3399