Re: [PATCH v2 0/6] automation: add QTB test framework for riscv64 smoke tests

["Alejandro Vallejo" <alejandro.garciavallejo@xxxxxxx>]

On Thu Aug 27, 2026 at 11:35 AM CEST, Baptiste Le Duc wrote:
> v1 was posted on 2026-08-10 with no response yet (13 business days). This
> v2 also fixes a bug flagged during internal review, see "Changes since v1"
> below.
>
> Xen is being made safety certifiable to IEC 61508 SIL 3 and ISO 26262 ASIL
> D, with Arm and x86 as the current targets [1]. Evidence at those levels is
> requirements based testing plus structural coverage, produced automatically
> and repeatably in CI.
>
> QTB (QEMU Test Bench) [2] is the framework AMD wrote for it as part of the
> Xen safety initiative. It drives a live QEMU instance over qtest, QMP and
> GDB, so a test has full access to the machine while emulation runs: read
> and write the consoles, inspect registers and memory, inject interrupts and
> faults, all from Python and all reproducible in a pipeline. QTB is not
> upstream in QEMU yet, but is planned to be.
>
> RISC-V is not in the certification scope today. It is also the youngest
> port, so it has close to no test infrastructure to undo, which makes it the
> cheapest place to adopt QTB. Starting the riscv64 tests on the framework
> now means the port grows its tests in the shape certification asks for as
> the port itself grows, instead of a pile of expect scripts to convert later
> if riscv64 ever becomes a certification target. It also puts a second
> architecture on QTB, which is useful to the framework itself before it is
> proposed to QEMU upstream.
>
> Concretely, the riscv64 CI coverage today is one expect script,
> automation/scripts/qemu-smoke-riscv64.sh, which boots Xen alone under QEMU
> and greps a single string out of one serial console. The machine it boots
> is hardcoded, so a second configuration means a second script, and a test
> with a DomU in it means growing guest handling from scratch.
>
> This series replaces that script with a data-driven framework using QTB. A
> machine is a YAML entry (pcpus, scheduler, interrupt controller, boot
> arguments), a test type is a small Python class saying what to do with a
> booted machine, and a test binds a machine to that type's expectations.
> Adding a configuration to CI is then a config change and a job stanza.
>
> Patch 1 goes to test-artifacts [3] and must land first, since patches 2-6
> run inside the container it adds. Patches 2-6 go to xen.git.
>
> test-artifacts [3]:
> - Add a QTB container to run the Xen riscv64 tests
>     debian:13-qtb-riscv64, carrying qemu.qtb from AMD's QEMU fork and the
>     dtc/fdt dependencies the framework needs at run time.
>
> xen.git:
> - automation/qtb: add jinja2 device trees for riscv64 smoke tests
>     The host tree varies with hart count, MMU type and device set, so it is
>     rendered per machine from a template rather than shipping one static
>     .dts per configuration.
> - automation/qtb: add Python QTB framework with the console-test type
>     The base layer every test type builds on (machine catalog lookup, host
>     DT generation, QEMU invocation, per-console log capture) plus the first
>     test type, which asserts every string listed in console-test.yaml is
>     printed on the expected console.
> - automation/qtb: add unit tests for the QTB framework
>     pytest coverage of the QEMU-agnostic parts, for developers only.
> - automation/qtb: add QTB framework README
>     How to add a machine, add a test type and run one locally.
> - CI: run the riscv64 smoke test via QTB framework console-test
>     Repoints qemu-smoke-riscv64-gcc at the framework and drops
>     qemu-smoke-riscv64.sh, which then has no caller left.
>
> Testing: Unit tests + QEMU only, as the existing riscv64 CI does.
> qemu-smoke-riscv64-gcc runs the same check as before.
>
> The catalog ships a single Xen-only machine, since Xen cannot boot a DomU
> on RISC-V yet. DomU machines plus an irq-test type using QTest interrupt
> injection follow once DomU support lands.
>
> CI pipeline:
> https://gitlab.com/xen-project/people/baptleduc/xen/-/pipelines/2795907042
>
> [1] 
> https://elisa.tech/blog/2026/07/22/the-final-phase-of-xen-safety-solving-coverage-and-residual-gaps-stefano-stabellini-amd/
> [2] https://gitlab.com/xen-project/people/amd/qemu/-/tree/safety
> [3] https://gitlab.com/xen-project/hardware/test-artifacts
> [4] 
> https://lore.kernel.org/xen-devel/1786980254.8631fc262581453bbf619ec5b2062170.1a01052c27d000c4f3@xxxxxxxxxx/

These patches are way too coarse to be looked at carefully, imo. It'd
help a lot if you managed to split further up the series until each
patch is <200LoC at least. Otherwise it's just a an exercise in
frustration just to find out how everything works.

Even at 200 it's annoying, but it's manageable at least.

Cheers,
Alejandro

>
> ---
> Changes since v1:
> - Resolve qemu-system-riscv64 from $PATH instead of pinning qemu-9.0.0 in
>   test.yaml, dropping the now-unneeded OpenSBI entry too: the
>   13-qtb-riscv64 container already bundles both (flagged by Zheng Zhang
>   during internal review [4]). Also drops the qemu-9.0.0-riscv64
>   test-artifacts dependency noted above.
>
> Baptiste Le Duc (5):
>   automation/qtb: add jinja2 device trees for riscv64 smoke tests
>   automation/qtb: add Python QTB framework with the console-test type
>   automation/qtb: add unit tests for the QTB framework
>   automation/qtb: add QTB framework README
>   CI: run the riscv64 smoke test via QTB framework console-test
>
>  .gitlab-ci.yml                                |   3 +
>  automation/gitlab-ci/test.yaml                |  20 +-
>  automation/scripts/qemu-smoke-riscv64.sh      |  19 --
>  automation/scripts/qemu_smoke_riscv64.py      | 122 ++++++++++
>  automation/scripts/qtb/__init__.py            |   2 +
>  automation/scripts/qtb/riscv/README.md        | 182 +++++++++++++++
>  automation/scripts/qtb/riscv/__init__.py      |   9 +
>  automation/scripts/qtb/riscv/config.py        | 119 ++++++++++
>  automation/scripts/qtb/riscv/config.yaml      |  17 ++
>  .../qtb/riscv/console_test/__init__.py        |   4 +
>  .../qtb/riscv/console_test/console-test.yaml  |  18 ++
>  .../qtb/riscv/console_test/console_test.py    | 145 ++++++++++++
>  automation/scripts/qtb/riscv/dt.py            |  57 +++++
>  .../scripts/qtb/riscv/dts/qemu-host.dts.j2    | 160 +++++++++++++
>  automation/scripts/qtb/riscv/machine.py       |  57 +++++
>  automation/scripts/qtb/riscv/paths.py         |  60 +++++
>  automation/scripts/qtb/riscv/qtb_test.py      |  53 +++++
>  automation/scripts/qtb/riscv/unit/__init__.py |   2 +
>  automation/scripts/qtb/riscv/unit/conftest.py |  42 ++++
>  .../scripts/qtb/riscv/unit/test_config.py     | 121 ++++++++++
>  .../qtb/riscv/unit/test_console_test.py       | 217 ++++++++++++++++++
>  automation/scripts/qtb/riscv/unit/test_dt.py  |  99 ++++++++
>  .../scripts/qtb/riscv/unit/test_machine.py    |  60 +++++
>  .../scripts/qtb/riscv/unit/test_temp_dir.py   |  42 ++++
>  .../scripts/qtb/riscv/unit/test_xen_dt.py     |  46 ++++
>  automation/scripts/qtb/riscv/xen_dt.py        |  58 +++++
>  26 files changed, 1709 insertions(+), 25 deletions(-)
>  delete mode 100755 automation/scripts/qemu-smoke-riscv64.sh
>  create mode 100755 automation/scripts/qemu_smoke_riscv64.py
>  create mode 100644 automation/scripts/qtb/__init__.py
>  create mode 100644 automation/scripts/qtb/riscv/README.md
>  create mode 100644 automation/scripts/qtb/riscv/__init__.py
>  create mode 100644 automation/scripts/qtb/riscv/config.py
>  create mode 100644 automation/scripts/qtb/riscv/config.yaml
>  create mode 100644 automation/scripts/qtb/riscv/console_test/__init__.py
>  create mode 100644 
> automation/scripts/qtb/riscv/console_test/console-test.yaml
>  create mode 100644 automation/scripts/qtb/riscv/console_test/console_test.py
>  create mode 100644 automation/scripts/qtb/riscv/dt.py
>  create mode 100644 automation/scripts/qtb/riscv/dts/qemu-host.dts.j2
>  create mode 100644 automation/scripts/qtb/riscv/machine.py
>  create mode 100644 automation/scripts/qtb/riscv/paths.py
>  create mode 100644 automation/scripts/qtb/riscv/qtb_test.py
>  create mode 100644 automation/scripts/qtb/riscv/unit/__init__.py
>  create mode 100644 automation/scripts/qtb/riscv/unit/conftest.py
>  create mode 100644 automation/scripts/qtb/riscv/unit/test_config.py
>  create mode 100644 automation/scripts/qtb/riscv/unit/test_console_test.py
>  create mode 100644 automation/scripts/qtb/riscv/unit/test_dt.py
>  create mode 100644 automation/scripts/qtb/riscv/unit/test_machine.py
>  create mode 100644 automation/scripts/qtb/riscv/unit/test_temp_dir.py
>  create mode 100644 automation/scripts/qtb/riscv/unit/test_xen_dt.py
>  create mode 100644 automation/scripts/qtb/riscv/xen_dt.py