Re: [PATCH 3/9] automation: Add debug versions of Arm tests

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Thu, 22 Sep 2022, Michal Orzel wrote:
> At the moment, all the tests are executed on non-debug Xen builds.
> To improve the coverage (e.g. we might catch some asserts), add new
> test jobs using debug Xen builds.

This is interesting. On one hand, I agree it is good to test DEBUG and
non-DEBUG builds in runtime tests too, on the other hand I am worried
about our test capacity.

I take you have tried a few pipelines -- are the additional tests
causing the pipeline to take longer? Do we still have "room" because x86
is slower? What's your thinking?

One idea would be to only do:
- qemu-alpine-arm64-gcc-debug
- qemu-smoke-arm64-gcc-debug

and not the others


> Take the opportunity to increase RAM size for QEMU from 1GB to 2GB
> in qemu-smoke-arm64.sh as the debug builds take more space and we might
> end up in a situation when there is not enough free space (especially
> during a static memory test that reserves some region in the middle).
> 
> Signed-off-by: Michal Orzel <michal.orzel@xxxxxxx>
> ---
>  automation/gitlab-ci/build.yaml        | 16 +++++
>  automation/gitlab-ci/test.yaml         | 93 ++++++++++++++++++++++++++
>  automation/scripts/qemu-smoke-arm64.sh |  6 +-
>  3 files changed, 112 insertions(+), 3 deletions(-)
> 
> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> index d931441b910a..6ce92531f61f 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -587,6 +587,15 @@ alpine-3.12-gcc-arm64-staticmem:
>        CONFIG_UNSUPPORTED=y
>        CONFIG_STATIC_MEMORY=y
>  
> +alpine-3.12-gcc-debug-arm64-staticmem:
> +  extends: .gcc-arm64-build-debug
> +  variables:
> +    CONTAINER: alpine:3.12-arm64v8
> +    EXTRA_XEN_CONFIG: |
> +      CONFIG_EXPERT=y
> +      CONFIG_UNSUPPORTED=y
> +      CONFIG_STATIC_MEMORY=y
> +
>  alpine-3.12-gcc-arm64-boot-cpupools:
>    extends: .gcc-arm64-build
>    variables:
> @@ -594,6 +603,13 @@ alpine-3.12-gcc-arm64-boot-cpupools:
>      EXTRA_XEN_CONFIG: |
>        CONFIG_BOOT_TIME_CPUPOOLS=y
>  
> +alpine-3.12-gcc-debug-arm64-boot-cpupools:
> +  extends: .gcc-arm64-build-debug
> +  variables:
> +    CONTAINER: alpine:3.12-arm64v8
> +    EXTRA_XEN_CONFIG: |
> +      CONFIG_BOOT_TIME_CPUPOOLS=y
> +
>  ## Test artifacts common
>  
>  .test-jobs-artifact-common:
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index 4f96e6e322de..1b51030c6175 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -44,6 +44,25 @@ qemu-alpine-arm64-gcc:
>    tags:
>      - arm64
>  
> +qemu-alpine-arm64-gcc-debug:
> +  extends: .test-jobs-common
> +  variables:
> +    CONTAINER: debian:unstable-arm64v8
> +  script:
> +    - ./automation/scripts/qemu-alpine-arm64.sh 2>&1 | tee 
> qemu-smoke-arm64.log
> +  needs:
> +    - alpine-3.12-gcc-debug-arm64
> +    - alpine-3.12-arm64-rootfs-export
> +    - kernel-5.19-arm64-export
> +    - qemu-system-aarch64-6.0.0-arm64-export
> +  artifacts:
> +    paths:
> +      - smoke.serial
> +      - '*.log'
> +    when: always
> +  tags:
> +    - arm64
> +
>  qemu-alpine-x86_64-gcc:
>    extends: .test-jobs-common
>    variables:
> @@ -81,6 +100,25 @@ qemu-smoke-arm64-gcc:
>    tags:
>      - arm64
>  
> +qemu-smoke-arm64-gcc-debug:
> +  extends: .test-jobs-common
> +  variables:
> +    CONTAINER: debian:unstable-arm64v8
> +  script:
> +    - ./automation/scripts/qemu-smoke-arm64.sh 2>&1 | tee 
> qemu-smoke-arm64.log
> +  needs:
> +    - alpine-3.12-gcc-debug-arm64
> +    - alpine-3.12-arm64-rootfs-export
> +    - kernel-5.19-arm64-export
> +    - qemu-system-aarch64-6.0.0-arm64-export
> +  artifacts:
> +    paths:
> +      - smoke.serial
> +      - '*.log'
> +    when: always
> +  tags:
> +    - arm64
> +
>  qemu-smoke-arm64-gcc-staticmem:
>    extends: .test-jobs-common
>    variables:
> @@ -100,6 +138,25 @@ qemu-smoke-arm64-gcc-staticmem:
>    tags:
>      - arm64
>  
> +qemu-smoke-arm64-gcc-debug-staticmem:
> +  extends: .test-jobs-common
> +  variables:
> +    CONTAINER: debian:unstable-arm64v8
> +  script:
> +    - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee 
> qemu-smoke-arm64.log
> +  needs:
> +    - alpine-3.12-gcc-debug-arm64-staticmem
> +    - alpine-3.12-arm64-rootfs-export
> +    - kernel-5.19-arm64-export
> +    - qemu-system-aarch64-6.0.0-arm64-export
> +  artifacts:
> +    paths:
> +      - smoke.serial
> +      - '*.log'
> +    when: always
> +  tags:
> +    - arm64
> +
>  qemu-smoke-arm64-gcc-boot-cpupools:
>    extends: .test-jobs-common
>    variables:
> @@ -119,6 +176,25 @@ qemu-smoke-arm64-gcc-boot-cpupools:
>    tags:
>      - arm64
>  
> +qemu-smoke-arm64-gcc-debug-boot-cpupools:
> +  extends: .test-jobs-common
> +  variables:
> +    CONTAINER: debian:unstable-arm64v8
> +  script:
> +    - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee 
> qemu-smoke-arm64.log
> +  needs:
> +    - alpine-3.12-gcc-debug-arm64-boot-cpupools
> +    - alpine-3.12-arm64-rootfs-export
> +    - kernel-5.19-arm64-export
> +    - qemu-system-aarch64-6.0.0-arm64-export
> +  artifacts:
> +    paths:
> +      - smoke.serial
> +      - '*.log'
> +    when: always
> +  tags:
> +    - arm64
> +
>  qemu-smoke-arm32-gcc:
>    extends: .test-jobs-common
>    variables:
> @@ -136,6 +212,23 @@ qemu-smoke-arm32-gcc:
>    tags:
>      - arm64
>  
> +qemu-smoke-arm32-gcc-debug:
> +  extends: .test-jobs-common
> +  variables:
> +    CONTAINER: debian:unstable-arm64v8
> +  script:
> +    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee 
> qemu-smoke-arm32.log
> +  needs:
> +    - debian-unstable-gcc-arm32-debug
> +    - qemu-system-aarch64-6.0.0-arm32-export
> +  artifacts:
> +    paths:
> +      - smoke.serial
> +      - '*.log'
> +    when: always
> +  tags:
> +    - arm64
> +
>  qemu-smoke-x86-64-gcc:
>    extends: .test-jobs-common
>    variables:
> diff --git a/automation/scripts/qemu-smoke-arm64.sh 
> b/automation/scripts/qemu-smoke-arm64.sh
> index f803835779f4..dea26c6c0a8f 100755
> --- a/automation/scripts/qemu-smoke-arm64.sh
> +++ b/automation/scripts/qemu-smoke-arm64.sh
> @@ -52,7 +52,7 @@ curl -fsSLO 
> https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
>  ./binaries/qemu-system-aarch64 \
>     -machine virtualization=true \
>     -cpu cortex-a57 -machine type=virt \
> -   -m 1024 -smp 2 -display none \
> +   -m 2048 -smp 2 -display none \
>     -machine dumpdtb=binaries/virt-gicv2.dtb
>  
>  # XXX disable pl061 to avoid Linux crash
> @@ -117,7 +117,7 @@ cd ..
>  
>  # ImageBuilder
>  echo 'MEMORY_START="0x40000000"
> -MEMORY_END="0x80000000"
> +MEMORY_END="0xC0000000"
>  
>  DEVICE_TREE="virt-gicv2.dtb"
>  XEN="xen"
> @@ -158,7 +158,7 @@ timeout -k 1 240 \
>  ./binaries/qemu-system-aarch64 \
>      -machine virtualization=true \
>      -cpu cortex-a57 -machine type=virt \
> -    -m 1024 -monitor none -serial stdio \
> +    -m 2048 -monitor none -serial stdio \
>      -smp 2 \
>      -no-reboot \
>      -device virtio-net-pci,netdev=n0 \

The changes to qemu-smoke-arm64.sh make sense, but could you move them
to a separate patch in case we need to do any backports?

Also I would change MEMORY_END to 0xC0000000 in qemu-alpine-arm64.sh too
for uniformity.