Xen project Mailing List

Sorry for all of the emails, but here is /proc/device-tree with XEN and without XEN:

with Xen:

With XEN

qos@ffa60080 interrupt-parent
qos@ffa90000 qos@ffaa8080
pmu-clock-controller@ff750000 watchdog@ff848000
pwm@ff420030 serial@ff1b0000
vcc5v0-usb-regulator qos@ffa76000
mmc@fe310000 iommu@ff903f00
i2c@ff110000 #size-cells
spdif@ff870000 qos@ffab0080
interrupt-controller qos@ffae0000
serial@ff370000 spi@ff200000
qos@ffac8080 xin24m
name hypervisor
vop@ff8f0000 psci
vcc12v-dcin vcc-0v9
clock-controller@ff760000 leds
qos@ffab8000 qos@ffaa0000
qos@ffa60100 iommu@ff914000
qos@ffac0000 model
i2c@ff3d0000 vcc5v0-sys
cpus video-codec@ff650000
mipi@ff968000 thermal-zones
isp0@ff910000 i2s@ff890000
external-gmac-clock rga@ff680000
dma-controller@ff6d0000 spi@ff1e0000
spi@ff1c0000 iommu@ff670800
qos@ffa5c000 dp@fec00000
iommu@ff650800 opp-table0
serial@ff190000 display-subsystem
vop@ff900000 qos@ffa98000
chosen i2c@ff160000
i2c@ff140000 qos@ffa70080
edp@ff970000 efuse@ff690000
opp-table1 syscon@ff770000
i2s@ff8a0000 hdmi-sound
usb@fe3c0000 mmc@fe320000
qos@ffa60180 qos@ffa58000
usb@fe800000 i2c@ff120000
qos@ffac0080 usb@fe3e0000
aliases qos@ffad8080
vcc3v3-pcie-regulator saradc@ff100000
qos@ffaa8000 usb@fe3a0000
phy@ff800000 opp-table2
hdmi@ff940000 qos@ffac8000
qos@ffab0000 qos@ffaa0080
i2c@ff3c0000 rktimer@ff850000
iommu@ff924000 qos@ffad0000
compatible pcie@f8000000
sdio-pwrseq i2c@ff3e0000
video-codec@ff660000 memory@20000000
timer gpu@ff9a0000
iommu@ff8f3f00 qos@ffa74000
mipi@ff960000 #address-cells
syscon@ff320000 power-management@ff310000
spi@ff1f0000 pinctrl
vcc3v3-sys pwm@ff420000
vcc-phy-regulator vcc1v8-s0
spi@ff350000 i2s@ff880000
ethernet@fe300000 dma-controller@ff6e0000
usb@fe380000 spi@ff1d0000
serial@ff180000 iommu@ff660480
usb@fe900000 pwm@ff420010
tsadc@ff260000 phy@ff7c0000
mmc@fe330000 pwm@ff420020
i2c@ff130000 i2c@ff150000
qos@ffa70000

Without Xen

root@zaku:/home/.cachengo# ls /proc/device-tree
'#address-cells' qos@ffa5c000
'#size-cells' qos@ffa60080
aliases qos@ffa60100
chosen qos@ffa60180
clock-controller@ff760000 qos@ffa70000
compatible qos@ffa70080
cpus qos@ffa74000
display-subsystem qos@ffa76000
dma-controller@ff6d0000 qos@ffa90000
dma-controller@ff6e0000 qos@ffa98000
dp@fec00000 qos@ffaa0000
edp@ff970000 qos@ffaa0080
efuse@ff690000 qos@ffaa8000
ethernet@fe300000 qos@ffaa8080
external-gmac-clock qos@ffab0000
gpu@ff9a0000 qos@ffab0080
hdmi-sound qos@ffab8000
hdmi@ff940000 qos@ffac0000
i2c@ff110000 qos@ffac0080
i2c@ff120000 qos@ffac8000
i2c@ff130000 qos@ffac8080
i2c@ff140000 qos@ffad0000
i2c@ff150000 qos@ffad8080
i2c@ff160000 qos@ffae0000
i2c@ff3c0000 rga@ff680000
i2c@ff3d0000 rktimer@ff850000
i2c@ff3e0000 saradc@ff100000
i2s@ff880000 sdio-pwrseq
i2s@ff890000 serial@ff180000
i2s@ff8a0000 serial@ff190000
interrupt-controller@fee00000 serial@ff1a0000
interrupt-parent serial@ff1b0000
iommu@ff650800 serial@ff370000
iommu@ff660480 spdif@ff870000
iommu@ff670800 spi@ff1c0000
iommu@ff8f3f00 spi@ff1d0000
iommu@ff903f00 spi@ff1e0000
iommu@ff914000 spi@ff1f0000
iommu@ff924000 spi@ff200000
isp0@ff910000 spi@ff350000
leds syscon@ff320000
memory syscon@ff770000
mipi@ff960000 thermal-zones
mipi@ff968000 timer
mmc@fe310000 tsadc@ff260000
mmc@fe320000 usb@fe380000
mmc@fe330000 usb@fe3a0000
model usb@fe3c0000
name usb@fe3e0000
opp-table0 usb@fe800000
opp-table1 usb@fe900000
opp-table2 vcc-0v9
pcie@f8000000 vcc-phy-regulator
phy@ff7c0000 vcc12v-dcin
phy@ff800000 vcc1v8-s0
pinctrl vcc3v3-pcie-regulator
pmu-clock-controller@ff750000 vcc3v3-sys
pmu_a53 vcc5v0-sys
pmu_a72 vcc5v0-usb-regulator
power-management@ff310000 video-codec@ff650000
psci video-codec@ff660000
pwm@ff420000 vop@ff8f0000
pwm@ff420010 vop@ff900000
pwm@ff420020 watchdog@ff848000
pwm@ff420030 xin24m
qos@ffa58000

On Tue, Jul 12, 2022 at 11:24 AM Brad Churchwell <brad@xxxxxxxxxxxx> wrote:

Hi Bertrand,

I believe I understand, but just to clarify, should I leave the ppi-partitions block in rk3399.dtsi as is and disable the little cores, or should I also modify that block?

Brad

On Tue, Jul 12, 2022 at 11:11 AM Bertrand Marquis <Bertrand.Marquis@xxxxxxx> wrote:
Hi Brad,

> On 12 Jul 2022, at 16:59, Brad Churchwell <brad@xxxxxxxxxxxx> wrote:
>
> Hi Bertrand,
>
> Thanks so much for the quick response!
>
> I should have mentioned previously that this device tree and kernel Image (5.15.16) does boot properly with the rootfs without XEN. The interrupt errors are only present when booting with XEN.
>
> These are custom boards and they do have usb c, however we are unable to boot from usb as it doesn't receive power. We currently only use the usb to flash u-boot to spi which requires us to use a separate power adapter. These boards get power and networking through a custom backplane and interface with the backplane via a sata style justice (I'm not sure what the proper terminology would be here).
>
> Since I cannot boot to rootfs with XEN, I'm unable to show the device tree there. The only bit that I added to accommodate XEN is this bit here:
>
> chosen {
> stdout-path = "serial2:1500000n8";
> bootargs = "hmp-unsafe=true";
> xen,dom0-bootargs = "console=hvc0 earlyprintk=xen clk_ignore_unused root=/dev/nvme0n1p2 rw init=/sbin/init rootwait rootfstype=ext4 ignore_loglevel";
> modules {
> #address-cells = <2>;
> #size-cells = <2>;
>
> module@1 {
> compatible = "xen,linux-zimage", "xen,multiboot-module";
> reg = <0x0 0x03F80000 0x0 0x01aa8008>;
> };
> module@2 {
> compatible = "xen,linux-initrd", "xen,multiboot-module";
> reg = <0x0 0x06000000 0x0 0x02000000>;
> };
> };
>
> };
>
> The gic declaration is in the vanilla rk3399.dtsi file and hasn't been modified:
>
> gic: interrupt-controller@fee00000 {
> compatible = "arm,gic-v3";
> #interrupt-cells = <4>;
> #address-cells = <2>;
> #size-cells = <2>;
> ranges;
> interrupt-controller;
>
> reg = <0x0 0xfee00000 0 0x10000>, /* GICD */
> <0x0 0xfef00000 0 0xc0000>, /* GICR */
> <0x0 0xfff00000 0 0x10000>, /* GICC */
> <0x0 0xfff10000 0 0x10000>, /* GICH */
> <0x0 0xfff20000 0 0x10000>; /* GICV */
> interrupts = <GIC_PPI 9 IRQ_TYPE_LEVEL_HIGH 0>;
> its: interrupt-controller@fee20000 {
> compatible = "arm,gic-v3-its";
> msi-controller;
> #msi-cells = <1>;
> reg = <0x0 0xfee20000 0x0 0x20000>;
> };
>
> ppi-partitions {
> ppi_cluster0: interrupt-partition-0 {
> affinity = <&cpu_l0 &cpu_l1 &cpu_l2 &cpu_l3>;
> };
>
> ppi_cluster1: interrupt-partition-1 {
> affinity = <&cpu_b0 &cpu_b1>;
> };
> };

Xen is not supporting PPI partitions.
I also saw that your booting using SMP unsafe so all core are enabled by Xen.

Right now big/little is not supported by Xen and here you might have some interrupts in your system which can only occur on some specific cores.
I would suggest to try the following:
- only boot enable in your dtb the cores with the same type than the boot core
- check in your device tree which devices are assign to the ppi_cluster not available and disable them

This kind of configuration will be tricky to handle so it will depend on which devices have interrupts that can only be fired on one cluster type.

Sorry but I cannot right now check in the device tree and do those checks myself but if you do get the idea send back a mail :-)

Cheers
Bertrand

> };
>
> I've attached the device tree and include files just in case you'd like to take a look at those.
>
> Thanks so much for your help! I've been at this for weeks with very little progress.
>
> Cheers,
> Brad
>
>
> On Tue, Jul 12, 2022 at 10:05 AM Bertrand Marquis <Bertrand.Marquis@xxxxxxx> wrote:
> Hi Brad,
>
> > On 11 Jul 2022, at 19:38, Brad Churchwell <brad@xxxxxxxxxxxx> wrote:
> >
> > Hello,
> >
> > I've been trying to get Xen to boot dom0 with my kernel for weeks on an rk3399 based board and thought I'd reach out for help. It looks like either Xen is not properly recreating the device tree or the interrupt controller is just failing. The hypervisor boots but falls to initramfs because it cannot find the root device (nvme on pcie). Any help would be greatly appreciated. Here is the complete boot log
>
> From the logs you have an issue with the interrupt controller and the fact that your NVME is behind PCIE and as such depends on ITS is probably not helping.
>
> I would suggest to try to boot on usb as root fs for a try.
>
> Also it could be useful to compare the device tree on xen and without xen to understand what is going on (using /proc/device-tree).
>
> Xen seems to be ok but Linux is not happy with interrupts and is showing several issues around this area on your logs.
> Could you show us an extract of your device tree around the gic declaration ?
>
> Cheers
> Bertrand
>
>
> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
> <rk3399-zaku.dts><rk3399-zaku.dtsi><rk3399.dtsi>

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: Xen on rk3399