Re: [Xen-devel] [PATCH v1 1/2] arm: Add ability to relocate Xen in over 4GB space

[Andrii Anisov <andrii.anisov@xxxxxxxxxxxxxxx>]

> OOI, can you give more context why you need to relocate Xen over 4GB?

Julien,

The context here is pretty simple: just a workaround of the real system limitations:

1. Renesas R-Car H2 evaluation board lager has 4GB of RAM, 2GB mapped under 4GB boundary and 2GB over.Â
   
2. The R-Car2 chip has IOMMU's for almost all active devices except GPU.Â
   
3. Within our system we need one user domain with 2GB of RAM and dedicated GPU.
   
4. While doing PoC we have no access to the GPU driver(and firmware) sources to introduce swiotlb here.Â
   

Having no option to handle GPU driver addressing with iommu (2) or swiotlb(4), it was decided to place the user domain described in 3 within RAM below 4GB with 1:1 mapping. Therefore we needed to move all the rest (XEN, rest of domains) under 4GB.

Andrii Anisov | Team Lead
GlobalLogic
Kyiv, 03038, Protasov Business Park, M.Grinchenka, 2/1
P +38.044.492.9695x3664 ÂM +380505738852 ÂS andriyanisov
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt


On Wed, Apr 8, 2015 at 7:05 PM, Julien Grall <julien.grall@xxxxxxxxxx> wrote:
> Hi Iurii,
>
> OOI, can you give more context why you need to relocate Xen over 4GB?
>
> On 08/04/15 13:36, Iurii Konovalenko wrote:
>> From: Iurii Konovalenko <iurii.konovalenko@xxxxxxxxxxxxxxx>
>>
>> Primary CPU relocate Xen in over 4GB space and wake up seondary CPUs.
>
> Typoes:
>
> s/relocate/relocates
> s/seondary/secondary
>
>> Secondary CPUs run on unrelocated copy of Xen until turning on MMU.
>> After turning on MMU secondary CPUs run on relocated copy of Xen.
>
> The non-relocated copy of Xen is not marked as reserved. So Xen may
> allocate the area for his own purpose before the secondary CPU has boot.
>
>>
>> To add ability to relocate Xen in over 4GB space add following to
>> compilation command: ARM32_RELOCATE_OVER_4GB=y
>
> The virtualization extension requires LPAE. Any reason to make this
> optional?
>
>> Signed-off-by: Iurii Konovalenko <iurii.konovalenko@xxxxxxxxxxxxxxx>
>> Signed-off-by: Andrii Anisov <andrii.anisov@xxxxxxxxxxxxxxx>
>> ---
>> Âxen/Rules.mk          | Â1 +
>> Âxen/arch/arm/arm32/head.S Â Â Â| 21 ++++++++++++++++++++-
>> Âxen/arch/arm/platforms/rcar2.c | Â9 ++++++++-
>> Âxen/arch/arm/setup.c      | 17 ++++++++++++++++-
>> Âxen/arch/arm/smpboot.c     | 33 +++++++++++++++++++++++++++++----
>> Â5 files changed, 74 insertions(+), 7 deletions(-)
>>
>> diff --git a/xen/Rules.mk b/xen/Rules.mk
>> index feb08d6..fbd34a5 100644
>> --- a/xen/Rules.mk
>> +++ b/xen/Rules.mk
>> @@ -64,6 +64,7 @@ CFLAGS-$(HAS_PCI) Â Â Â += -DHAS_PCI
>> ÂCFLAGS-$(HAS_IOPORTS) Â += -DHAS_IOPORTS
>> ÂCFLAGS-$(HAS_PDX) Â Â Â += -DHAS_PDX
>> ÂCFLAGS-$(frame_pointer) += -fno-omit-frame-pointer -DCONFIG_FRAME_POINTER
>> +CFLAGS-$(ARM32_RELOCATE_OVER_4GB) += -DARM32_RELOCATE_OVER_4GB
>>
>> Âifneq ($(max_phys_cpus),)
>> ÂCFLAGS-y        Â+= -DMAX_PHYS_CPUS=$(max_phys_cpus)
>> diff --git a/xen/arch/arm/arm32/head.S b/xen/arch/arm/arm32/head.S
>> index 5c0263e..26be1d0 100644
>> --- a/xen/arch/arm/arm32/head.S
>> +++ b/xen/arch/arm/arm32/head.S
>> @@ -262,8 +262,21 @@ cpu_init_done:
>>     Âadd  r4, r4, r10      Â/* r4 := paddr (boot_pagetable) */
>>     Âmov  r5, #0         /* r4:r5 is paddr (boot_pagetable) */
>> Â Â Â Â Âmcrr ÂCP64(r4, r5, HTTBR)
>
> Newline here.
>
>> +#ifdef ARM32_RELOCATE_OVER_4GB
>> +    Âteq  r7, #0
>> +    Âbeq  1f           /* construct pagetable if CPU0 */
>>
>> - Â Â Â Â/* Setup boot_pgtable: */
>> + Â Â Â Â/*Skip constructing TLBs for secondary CPUs, use constructed by CPU0*/
>
> /* ... */
>
> TLBs is confusing. I first though you were talking to Translation
> Lookaside Buffer.
>
>> + Â Â Â ÂPRINT("- Skip construction pagetable, using CPU0 table @")
>
> This very fragile. The CPU0 may drop a part of the 1:1 mapping if it's
> clashes with the fixmap and DTB mappings (see after the label "paging").
>
> Therefore secondary CPUs may hang during boot.
>
>> +    Âmov  r0, r5
>> +    Âbl  Âputn
>> +    Âmov  r0, r4
>> +    Âbl  Âputn
>> + Â Â Â ÂPRINT(" Â-\r\n")
>> +    Âb  skip_constructing
>> +#endif
>> +
>> +1: Â Â Â/* Setup boot_pgtable: */
>>     Âldr  r1, =boot_second
>>     Âadd  r1, r1, r10      Â/* r1 := paddr (boot_second) */
>>
>> @@ -346,6 +359,7 @@ virtphys_clash:
>> Â Â Â Â ÂPRINT("- Unable to build boot page tables - virt and phys addresses clash. -\r\n")
>>     Âb   fail
>>
>> +skip_constructing:
>> Â1:
>> Â Â Â Â ÂPRINT("- Turning on paging -\r\n")
>>
>> @@ -427,6 +441,11 @@ paging:
>> Â Â Â Â Â * setup in init_secondary_pagetables. */
>>
>>     Âldr  r4, =init_ttbr     /* VA of HTTBR value stashed by CPU 0 */
>> +#ifdef ARM32_RELOCATE_OVER_4GB
>> +    Âldr  r1, =_start
>> + Â Â Â Âsub r4, r1
>> + Â Â Â Âadd r4, #BOOT_RELOC_VIRT_START
>> +#endif
>
> This chunk need some comment in order to explain what it's doing.
>
> AFAIU the resulting value in r4, will be exactly the same as Â"ldr r4,
> =init_ttbr".
>
>> Â Â Â Â Âldrd Âr4, r5, [r4] Â Â Â Â Â /* Actual value */
>> Â Â Â Â Âdsb
>> Â Â Â Â Âmcrr ÂCP64(r4, r5, HTTBR)
>> diff --git a/xen/arch/arm/platforms/rcar2.c b/xen/arch/arm/platforms/rcar2.c
>> index aef544c..4365166 100644
>> --- a/xen/arch/arm/platforms/rcar2.c
>> +++ b/xen/arch/arm/platforms/rcar2.c
>> @@ -25,6 +25,9 @@
>> Â#define RCAR2_RAM_SIZE Â Â Â Â Â Â Â Â Â Â Â Â 0x1000
>> Â#define RCAR2_SMP_START_OFFSET Â Â Â Â Â Â Â Â 0xFFC
>>
>> +#ifdef ARM32_RELOCATE_OVER_4GB
>> +extern paddr_t xen_relocation_offset;
>> +#endif
>
> Please define in an header.
>
>> Âstatic int __init rcar2_smp_init(void)
>> Â{
>> Â Â Âvoid __iomem *pram;
>> @@ -38,7 +41,11 @@ static int __init rcar2_smp_init(void)
>> Â Â Â}
>>
>> Â Â Â/* setup reset vectors */
>> - Â Âwritel(__pa(init_secondary), pram + RCAR2_SMP_START_OFFSET);
>> + Â Âwritel(__pa(init_secondary)
>> +#ifdef ARM32_RELOCATE_OVER_4GB
>> + Â Â Â Â Â Â- xen_relocation_offset
>> +#endif
>> + Â Â Â Â Â Â, pram + RCAR2_SMP_START_OFFSET);
>> Â Â Âiounmap(pram);
>>
>> Â Â Âsev();
>> diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
>> index 4ec7c13..66e2834 100644
>> --- a/xen/arch/arm/setup.c
>> +++ b/xen/arch/arm/setup.c
>> @@ -407,7 +407,7 @@ static paddr_t __init get_xen_paddr(void)
>> Â Â Â Â Â Â Âif ( !e )
>> Â Â Â Â Â Â Â Â Âcontinue;
>>
>> -#ifdef CONFIG_ARM_32
>> +#if defined (CONFIG_ARM_32) && !defined(ARM32_RELOCATE_OVER_4GB)
>> Â Â Â Â Â Â Â/* Xen must be under 4GB */
>> Â Â Â Â Â Â Âif ( e > 0x100000000ULL )
>> Â Â Â Â Â Â Â Â Âe = 0x100000000ULL;
>> @@ -699,6 +699,9 @@ void __init setup_cache(void)
>> Â Â Âcacheline_bytes = 1U << (4 + (ccsid & 0x7));
>> Â}
>>
>> +#ifdef ARM32_RELOCATE_OVER_4GB
>> +paddr_t xen_relocation_offset;
>> +#endif
>> Â/* C entry point for boot CPU */
>> Âvoid __init start_xen(unsigned long boot_phys_offset,
>> Â Â Â Â Â Â Â Â Â Â Â Âunsigned long fdt_paddr,
>> @@ -740,8 +743,20 @@ void __init start_xen(unsigned long boot_phys_offset,
>> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â (paddr_t)(uintptr_t)(_end - _start + 1), NULL);
>> Â Â ÂBUG_ON(!xen_bootmodule);
>>
>> +#ifdef ARM32_RELOCATE_OVER_4GB
>> + Â Â//save physical address of init_secondary
>
> Xen is using /* ... */ for the comment.
>
>> + Â Âxen_relocation_offset = __pa(init_secondary);
>> +#endif
>> Â Â Âxen_paddr = get_xen_paddr();
>> Â Â Âsetup_pagetables(boot_phys_offset, xen_paddr);
>> +#ifdef ARM32_RELOCATE_OVER_4GB
>> + Â Â//Now Xen is relocated
>> + Â Â//Calculate offset of Xen relocation
>> + Â Â//It is difference between new physical address of init_secondary an old one
>> + Â Â//This offset is needed in several places when we have to write to old Xen location
>> + Â Â//(secondary CPUs run on old-located Xen and rely on some variables from CPU0)
>
> Ditto
>
>> + Â Âxen_relocation_offset = __pa(init_secondary) - xen_relocation_offset;
>
> The Xen may be relocated below the boot copy. So the final result may be
> negative.
>
>> +#endif
>>
>> Â Â Â/* Update Xen's address now that we have relocated. */
>> Â Â Âprintk("Update BOOTMOD_XEN from %"PRIpaddr"-%"PRIpaddr" => %"PRIpaddr"-%"PRIpaddr"\n",
>> diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
>> index a96cda2..731144c 100644
>> --- a/xen/arch/arm/smpboot.c
>> +++ b/xen/arch/arm/smpboot.c
>> @@ -31,6 +31,9 @@
>> Â#include <xen/console.h>
>> Â#include <asm/gic.h>
>> Â#include <asm/psci.h>
>> +#ifdef ARM32_RELOCATE_OVER_4GB
>> +#include <xen/vmap.h>
>> +#endif
>>
>> Âcpumask_t cpu_online_map;
>> Âcpumask_t cpu_present_map;
>> @@ -353,17 +356,33 @@ int __init cpu_up_send_sgi(int cpu)
>> Â Â Âreturn 0;
>> Â}
>>
>> +#ifdef ARM32_RELOCATE_OVER_4GB
>> +extern paddr_t xen_relocation_offset;
>> +#endif
>
> This can be avoid by defining xen_relocation_offset in an header.
>
>> Â/* Bring up a remote CPU */
>> Âint __cpu_up(unsigned int cpu)
>> Â{
>> Â Â Âint rc;
>> Â Â Âs_time_t deadline;
>> +#ifdef ARM32_RELOCATE_OVER_4GB
>> + Â Âpaddr_t p_info = __pa(&smp_up_cpu) - xen_relocation_offset;
>> + Â Âunsigned long* info = ioremap_nocache(p_info, sizeof(unsigned long));
>
> unsigned long *info and sizeof(smp_up_cpu).
>
> ioremap_nocache may fail and "info" doesn't have a real meaning.
>
> Regards,
>
> --
> Julien Grall
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel