Xen project Mailing List

Re: [Xen-devel] ARM64:Porting xen to new hardware

From: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>

Date: Fri, 22 Sep 2017 09:43:12 -0400

Cc: Oleksandr Tyshchenko <olekstysh@xxxxxxxxx>, Julien Grall <julien.grall@xxxxxxx>, Stefano Stabellini <sstabellini@xxxxxxxxxx>, Xen Devel <xen-devel@xxxxxxxxxxxxx>

Delivery-date: Fri, 22 Sep 2017 13:43:28 +0000

List-id: Xen developer discussion <xen-devel.lists.xen.org>

On Fri, Sep 22, 2017 at 03:30:57PM +0530, bharat gohil wrote: > Hello Wilk, > > I had try 'console=hvc0' but no success. > > @Oleksandr: At this moment it is difficult to share any file of guest. > It would be helpful if anyone provide me general technique to debug dom0 > bringup issue. You can also hit 'Ctrl-a' three times and then 'd' which would give you the stack trace and EIP of dom0. That should help in figuirng where your dom0 is stuck. > > Thanks, > Bharat > > On Mon, Sep 18, 2017 at 8:16 PM, Konrad Rzeszutek Wilk < > konrad.wilk@xxxxxxxxxx> wrote: > > > On Fri, Sep 08, 2017 at 10:19:55PM +0300, Oleksandr Tyshchenko wrote: > > > Hi Bharat > > > > > > On Thu, Sep 7, 2017 at 4:30 PM, bharat gohil <ghl.bhrt@xxxxxxxxx> wrote: > > > > Hello Olensandr, > > > > > > > > I able to boot xen and trying to boot dom0 but there are no console > > log for > > > > dom0. > > > > > > > > following log for xen and it stuck booting dom0. > > > > > > > > (XEN) I/O virtualisation disabled > > > > (XEN) build-id: 7c2a3c70fb94754801d18c4cb9e3db3ffa01d8c4 > > > > (XEN) alternatives: Patching with alt table 00000000400d2e08 -> > > > > 00000000400d32dc > > > > (XEN) *** LOADING DOMAIN 0 *** > > > > (XEN) Loading kernel from boot module @ 0000000040148158 > > > > (XEN) Allocating 1:1 mappings totalling 128MB for dom0: > > > > (XEN) BANK[0] 0x00000048000000-0x00000050000000 (128MB) > > > > (XEN) Grant table range: 0x000000bfe00000-0x000000bfe65000 > > > > (XEN) Loading zImage from 0000000040148158 to > > > > 0000000048080000-0000000049480000 > > > > (XEN) Allocating PPI 16 for event channel interrupt > > > > (XEN) Loading dom0 DTB to 0x000000004fe00000-0x000000004fe0f31e > > > > (XEN) Scrubbing Free RAM on 1 nodes using 3 CPUs > > > > (XEN) ......done. > > > > (XEN) Initial low memory virq threshold set at 0x4000 pages. > > > > (XEN) Std. Loglevel: All > > > > (XEN) Guest Loglevel: All > > > > (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch > > input to > > > > Xen) > > > > (XEN) Freed 272kB init memory. > > > > > > > > I have done all the xen configuration in linux kernel 4.9. This kernel > > > > booting fine without xen. > > > > > > > > following are the DTB changes, > > > > > > > > chosen { > > > > #address-cells = <1>; > > > > #size-cells = <1>; > > > > bootargs = "console=dtuart dtuart=serial0 dom0_mem=128M"; > > > > stdout-path = "serial0"; > > > > module: module@0 { > > > > compatible = "xen,linux-zimage", "xen,multiboot-module"; > > > > reg = <0x40148158 0x1400000>; > > > > bootargs = "console=hvc0,921600n8 earlyprintk=xen debug > > > > It should be just 'console=hvc0', not 'console=hvc0,921600n8' > > > > > > ignore_loglevel rw root=/dev/mmcblk0p7"; > > > > }; > > > > > > > > }; > > > > > > > > Can you tell me how to debug dom0 booting or anything which i can > > check? > > > > > > Don't now much about "debug dom0 booting", I leave it for competent > > people. > > > > > > Looks weird, even with earlyprintk no logs. > > > Do you have DEBUG_LL and all related options enabled in your dom0 kernel > > config? > > > > > > 1. Check that following options are enabled in your kernel config file: > > > > > > CONFIG_HVC_XEN=y > > > CONFIG_HVC_XEN_FRONTEND=y > > > > > > 2. Check that dom0 kernel doesn't disable clock for console. > > > > > > BTW, could you post full Xen log, kernel config and device-tree you are > > using? > > > If you have some changes on top of Xen, post them too. > > > These may help people to identify what is wrong. > > > > > > > > > > > > > > > Thanks, > > > > Bharat > > > > > > > > On Wed, Sep 6, 2017 at 3:49 PM, Oleksandr Tyshchenko < > > olekstysh@xxxxxxxxx> > > > > wrote: > > > >> > > > >> Hi Bharat > > > >> > > > >> On Wed, Sep 6, 2017 at 10:01 AM, bharat gohil <ghl.bhrt@xxxxxxxxx> > > wrote: > > > >> > Hello Oleksandr, > > > >> > > > > >> > Thank you very much.It resolved my issue. > > > >> Sounds great! > > > >> > > > >> > > > > >> > Thanks, > > > >> > Bharat > > > >> > > > > >> > On Mon, Sep 4, 2017 at 6:24 PM, Oleksandr Tyshchenko > > > >> > <olekstysh@xxxxxxxxx> > > > >> > wrote: > > > >> >> > > > >> >> Hi Bharat > > > >> >> > > > >> >> On Mon, Sep 4, 2017 at 7:13 AM, bharat gohil <ghl.bhrt@xxxxxxxxx> > > > >> >> wrote: > > > >> >> > Hello Oleksandr, > > > >> >> > > > > >> >> > I have corrected GIC settings but no success.Following line > > > >> >> > disappear > > > >> >> > from > > > >> >> > log. > > > >> >> >>>XEN) GICv2: WARNING: The GICC size is too small: 0x1000 expected > > > >> >> >>> 0x2000 > > > >> >> > > > > >> >> > Is anything else which can I try. > > > >> >> > > > > >> >> > I don’t know much about xen internal for ARM architecture. As you > > > >> >> > mentioned, > > > >> >> >>>Wrong GIC settings might lead to that IPIs won't work as > > expected. > > > >> >> >>> And > > > >> >> >>>boot CPU will get stuck waiting for another CPU. > > > >> >> > > > > >> >> > Can you explain it with some boot sequence and relation with IPI? > > > >> >> > > > >> >> Well, we faced similar issue with R-Car Gen3 H3 SoC. Xen hung at > > > >> >> smp_call_function (one CPU didn't receive interrupt from another > > one). > > > >> >> Next patch helped us to fix this issue: > > > >> >> https://patchwork.kernel.org/patch/9163065/ > > > >> >> > > > >> >> I assume the SoC you are working with has "arm,gic-400" compatible > > GIC. > > > >> >> Can you take a look at the patch, maybe it is your case too. > > > >> >> > > > >> >> > > > > >> >> > Thanks, > > > >> >> > Bharat > > > >> >> > > > > >> >> > > > > >> >> > On Thu, Aug 31, 2017 at 5:28 PM, Oleksandr Tyshchenko > > > >> >> > <olekstysh@xxxxxxxxx> > > > >> >> > wrote: > > > >> >> >> > > > >> >> >> On Thu, Aug 31, 2017 at 2:13 PM, bharat gohil < > > ghl.bhrt@xxxxxxxxx> > > > >> >> >> wrote: > > > >> >> >> > Hello Oleksandr, > > > >> >> >> Hi Bharat > > > >> >> >> > > > >> >> >> > > > > >> >> >> > I had removed A72 cluster and tried to boot only two A35 but > > I got > > > >> >> >> > same > > > >> >> >> > error. > > > >> >> >> > > > > >> >> >> > Is anything added or missing in A35 compare to A53? > > > >> >> >> Unfortunately, I don't know. > > > >> >> >> > > > >> >> >> BTW, did you check your GIC settings in the device-tree? > > > >> >> >> > > > >> >> >> > > > > >> >> >> > Regards, > > > >> >> >> > Bharat > > > >> >> >> > > > > >> >> >> > On Wed, Aug 30, 2017 at 8:00 PM, bharat gohil < > > ghl.bhrt@xxxxxxxxx> > > > >> >> >> > wrote: > > > >> >> >> >> > > > >> >> >> >> Hello Oleksandr, > > > >> >> >> >> Thank you very much for your input. > > > >> >> >> >> > > > >> >> >> >> Yes. agree. I will check by removing A72 core from DT. > > > >> >> >> >> > > > >> >> >> >> Thanks, > > > >> >> >> >> Bharat > > > >> >> >> >> > > > >> >> >> >> On Wed, Aug 30, 2017 at 7:44 PM, Oleksandr Tyshchenko > > > >> >> >> >> <olekstysh@xxxxxxxxx> wrote: > > > >> >> >> >>> > > > >> >> >> >>> Hi, > > > >> >> >> >>> > > > >> >> >> >>> Not sure that I am a competent person, just my assumptions. > > > >> >> >> >>> > > > >> >> >> >>> CCed ARM guys. > > > >> >> >> >>> > > > >> >> >> >>> On Tue, Aug 29, 2017 at 5:21 PM, bharat gohil > > > >> >> >> >>> <ghl.bhrt@xxxxxxxxx> > > > >> >> >> >>> wrote: > > > >> >> >> >>> > Hello All > > > >> >> >> >>> > > > > >> >> >> >>> > I am trying to run Xen on new hardware which has two A35 > > and > > > >> >> >> >>> > one > > > >> >> >> >>> > A72 > > > >> >> >> >>> > core. > > > >> >> >> >>> > Xen booted intially but it hangs at > > > >> >> >> >>> > smp_call_function(setup_virt_paging_one, > > > >> >> >> >>> > (void *)val, 1) function call. > > > >> >> >> >>> > > > >> >> >> >>> It might be a consequence of that CPU cores are different. > > And > > > >> >> >> >>> they > > > >> >> >> >>> might have different set of features, or even settings. > > > >> >> >> >>> And these features/settings the boot CPU has don't > > compatible > > > >> >> >> >>> with > > > >> >> >> >>> other (non-boot) CPUs. > > > >> >> >> >>> Can you try not to bringup A72 core (remove it from DT or > > > >> >> >> >>> another > > > >> >> >> >>> way), leave only two A35 and see what will happen. > > > >> >> >> >>> > > > >> >> >> >>> > Find following log of Xen booting,same set of features. > > > >> >> >> >>> > > > > >> >> >> >>> > - UART enabled - > > > >> >> >> >>> > - CPU 00000000 booting - > > > >> >> >> >>> > - Current EL 00000008 - > > > >> >> >> >>> > - Xen starting at EL2 - > > > >> >> >> >>> > - Zero BSS - > > > >> >> >> >>> > - Setting up control registers - > > > >> >> >> >>> > - Turning on paging - > > > >> >> >> >>> > - Ready - > > > >> >> >> >>> > (XEN) Checking for initrd in /chosen > > > >> >> >> >>> > (XEN) RAM: 0000000040000000 - 00000000bfffffff > > > >> >> >> >>> > (XEN) > > > >> >> >> >>> > (XEN) MODULE[0]: 0000000044000000 - 000000004400fd5a > > Device > > > >> >> >> >>> > Tree > > > >> >> >> >>> > (XEN) > > > >> >> >> >>> > (XEN) Command line: <NULL> > > > >> >> >> >>> Why? Does your device-tree have bootargs? > > > >> >> >> >>> > > > >> >> >> >>> > (XEN) Placing Xen at 0x00000000bfe00000- > > 0x00000000c0000000 > > > >> >> >> >>> > (XEN) Update BOOTMOD_XEN from > > > >> >> >> >>> > 0000000040080000-0000000040194e01 > > > >> >> >> >>> > => > > > >> >> >> >>> > 00000000bfe01 > > > >> >> >> >>> > (XEN) Domain heap initialised > > > >> >> >> >>> > (XEN) Booting using Device Tree > > > >> >> >> >>> > (XEN) Platform: Generic System > > > >> >> >> >>> > (XEN) Taking dtuart configuration from /chosen/stdout-path > > > >> >> >> >>> > (XEN) Looking for dtuart at "serial0", options "" > > > >> >> >> >>> > __ __ _ _ _ ___ _ > > > >> >> >> >>> > _ > > > >> >> >> >>> > _ > > > >> >> >> >>> > \ \/ /___ _ __ | || | / |/ _ \ _ _ _ __ ___| |_ > > __ > > > >> >> >> >>> > _| > > > >> >> >> >>> > |__ > > > >> >> >> >>> > | | > > > >> >> >> >>> > ___ > > > >> >> >> >>> > \ // _ \ '_ \ | || |_ | | | | |__| | | | '_ \/ __| > > __/ _` > > > >> >> >> >>> > | > > > >> >> >> >>> > '_ > > > >> >> >> >>> > \| > > > >> >> >> >>> > |/ _ \ > > > >> >> >> >>> > / \ __/ | | | |__ _|| | |_| |__| |_| | | | \__ \ || > > (_| > > > >> >> >> >>> > | > > > >> >> >> >>> > |_) > > > >> >> >> >>> > | | > > > >> >> >> >>> > __/ > > > >> >> >> >>> > /_/\_\___|_| |_| |_|(_)_|\___/ \__,_|_| > > > >> >> >> >>> > |_|___/\__\__,_|_.__/|_|\___| > > > >> >> >> >>> > > > > >> >> >> >>> > (XEN) Xen version 4.10-unstable (bgohil@) > > > >> >> >> >>> > (aarch64-linux-gnu-gcc > > > >> >> >> >>> > (Ubuntu/Linaro7 > > > >> >> >> >>> > (XEN) Latest ChangeSet: Fri Aug 11 19:02:51 2017 +0100 > > > >> >> >> >>> > git:9053a74-dirty > > > >> >> >> >>> > (XEN) Processor: 410fd041: "ARM Limited", variant: 0x0, > > part > > > >> >> >> >>> > 0xd04, > > > >> >> >> >>> > rev > > > >> >> >> >>> > 0x1 > > > >> >> >> >>> > (XEN) 64-bit Execution: > > > >> >> >> >>> > (XEN) Processor Features: 0000000000002222 > > 0000000000000000 > > > >> >> >> >>> > (XEN) Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 > > > >> >> >> >>> > EL0:64+32 > > > >> >> >> >>> > (XEN) Extensions: FloatingPoint AdvancedSIMD > > > >> >> >> >>> > (XEN) Debug Features: 0000000010305106 0000000000000000 > > > >> >> >> >>> > (XEN) Auxiliary Features: 0000000000000000 > > 0000000000000000 > > > >> >> >> >>> > (XEN) Memory Model Features: 0000000000101122 > > > >> >> >> >>> > 0000000000000000 > > > >> >> >> >>> > (XEN) ISA Features: 0000000000011120 0000000000000000 > > > >> >> >> >>> > (XEN) 32-bit Execution: > > > >> >> >> >>> > (XEN) Processor Features: 00000131:00011011 > > > >> >> >> >>> > (XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 > > Jazelle > > > >> >> >> >>> > (XEN) Extensions: GenericTimer Security > > > >> >> >> >>> > (XEN) Debug Features: 03010066 > > > >> >> >> >>> > (XEN) Auxiliary Features: 00000000 > > > >> >> >> >>> > (XEN) Memory Model Features: 10201105 40000000 01260000 > > > >> >> >> >>> > 02102211 > > > >> >> >> >>> > (XEN) ISA Features: 02101110 13112111 21232042 01112131 > > > >> >> >> >>> > 00011142 > > > >> >> >> >>> > 00011121 > > > >> >> >> >>> > (XEN) Using PSCI-1.0 for SMP bringup > > > >> >> >> >>> > (XEN) SMP: Allowing 3 CPUs > > > >> >> >> >>> > (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: > > 13000 > > > >> >> >> >>> > KHz > > > >> >> >> >>> > (XEN) GICv2: WARNING: The GICC size is too small: 0x1000 > > > >> >> >> >>> > expected > > > >> >> >> >>> > 0x2000 > > > >> >> >> >>> Sounds like GIC settings are not completely correct. > > > >> >> >> >>> Wrong GIC settings might lead to that IPIs won't work as > > > >> >> >> >>> expected. > > > >> >> >> >>> And > > > >> >> >> >>> boot CPU will > > > >> >> >> >>> get stuck waiting for another CPU. > > > >> >> >> >>> Just double check. > > > >> >> >> >>> > > > >> >> >> >>> > (XEN) GICv2 initialization: > > > >> >> >> >>> > (XEN) gic_dist_addr=0000000010510000 > > > >> >> >> >>> > (XEN) gic_cpu_addr=0000000010520000 > > > >> >> >> >>> > (XEN) gic_hyp_addr=0000000010540000 > > > >> >> >> >>> > (XEN) gic_vcpu_addr=0000000010560000 > > > >> >> >> >>> > (XEN) gic_maintenance_irq=25 > > > >> >> >> >>> > (XEN) GICv2: 384 lines, 6 cpus, secure (IID 0200143b). > > > >> >> >> >>> > (XEN) Using scheduler: SMP Credit Scheduler (credit) > > > >> >> >> >>> > (XEN) Allocated console ring of 32 KiB. > > > >> >> >> >>> > (XEN) Bringing up CPU1 > > > >> >> >> >>> > - CPU 00000001 booting - > > > >> >> >> >>> > - Current EL 00000008 - > > > >> >> >> >>> > - Xen starting at EL2 - > > > >> >> >> >>> > - Setting up control registers - > > > >> >> >> >>> > - Turning on paging - > > > >> >> >> >>> > - Ready - > > > >> >> >> >>> > (XEN) CPU 1 booted. > > > >> >> >> >>> > (XEN) Bringing up CPU2 > > > >> >> >> >>> > - CPU 00000200 booting - > > > >> >> >> >>> > - Current EL 00000008 - > > > >> >> >> >>> > - Xen starting at EL2 - > > > >> >> >> >>> > - Setting up control registers - > > > >> >> >> >>> > - Turning on paging - > > > >> >> >> >>> > - Ready - > > > >> >> >> >>> > (XEN) CPU 2 booted. > > > >> >> >> >>> > (XEN) Brought up 3 CPUs > > > >> >> >> >>> > (XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit VMID > > > >> >> >> >>> > (XEN) P2M: 3 levels with order-1 root, VTCR 0x80023558 > > > >> >> >> >>> > > > > >> >> >> >>> > Can anyone guide me how to debug this problem or what > > could be > > > >> >> >> >>> > wrong > > > >> >> >> >>> > here? > > > >> >> >> >>> > > > > >> >> >> >>> > It looks, writing into VTCR_EL2 hang the system. > > > >> >> >> >>> > > > > >> >> >> >>> > -- > > > >> >> >> >>> > Regards, > > > >> >> >> >>> > Bharat Gohil > > > >> >> >> >>> > > > > >> >> >> >>> > > > > >> >> >> >>> > _______________________________________________ > > > >> >> >> >>> > Xen-devel mailing list > > > >> >> >> >>> > Xen-devel@xxxxxxxxxxxxx > > > >> >> >> >>> > https://lists.xen.org/xen-devel > > > >> >> >> >>> > > > > >> >> >> >>> > > > >> >> >> >>> -- > > > >> >> >> >>> Regards, > > > >> >> >> >>> > > > >> >> >> >>> Oleksandr Tyshchenko > > > >> >> >> >> > > > >> >> >> >> > > > >> >> >> >> > > > >> >> >> >> > > > >> >> >> >> -- > > > >> >> >> >> Regards, > > > >> >> >> >> Bharat Gohil > > > >> >> >> >> Sr.Software Engineer > > > >> >> >> >> bharat.gohil@xxxxxxxxxx > > > >> >> >> >> +919427054633 > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > -- > > > >> >> >> > Regards, > > > >> >> >> > Bharat Gohil > > > >> >> >> > Sr.Software Engineer > > > >> >> >> > bharat.gohil@xxxxxxxxxx > > > >> >> >> > +919427054633 > > > >> >> >> > > > >> >> >> > > > >> >> >> > > > >> >> >> -- > > > >> >> >> Regards, > > > >> >> >> > > > >> >> >> Oleksandr Tyshchenko > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > -- > > > >> >> > Regards, > > > >> >> > Bharat Gohil > > > >> >> > Sr.Software Engineer > > > >> >> > bharat.gohil@xxxxxxxxxx > > > >> >> > +919427054633 > > > >> >> > > > >> >> > > > >> >> > > > >> >> -- > > > >> >> Regards, > > > >> >> > > > >> >> Oleksandr Tyshchenko > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > -- > > > >> > Regards, > > > >> > Bharat Gohil > > > >> > Sr.Software Engineer > > > >> > bharat.gohil@xxxxxxxxxx > > > >> > +919427054633 > > > >> > > > >> > > > >> > > > >> -- > > > >> Regards, > > > >> > > > >> Oleksandr Tyshchenko > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > Bharat Gohil > > > > Sr.Software Engineer > > > > bharat.gohil@xxxxxxxxxx > > > > +919427054633 > > > > > > > > > > > > -- > > > Regards, > > > > > > Oleksandr Tyshchenko > > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@xxxxxxxxxxxxx > > > https://lists.xen.org/xen-devel > > > > > > -- > Regards, > Bharat Gohil > Sr.Software Engineer > bharat.gohil@xxxxxxxxxx > +919427054633 _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx https://lists.xen.org/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.