[PATCH v3 3/6] automation/qtb: add the console-test type with a Xen boot test

[Baptiste Le Duc <baptiste.le-duc@xxxxxxxxxx>]

Add a first console test, which boots Xen under QEMU and waits for "All set
up" on its console. It is the equivalent of qemu-smoke-riscv64.sh, with the
finer control QTB gives, and new configurations are added as test
parameters instead of new scripts.

The tests get their machine through the `machine` pytest fixture. A test
parametrizes the fixture indirectly with the pCPU count, and the fixture
builds the matching RiscvTestMachine from it, so a new configuration only
takes a pytest.param() above the test function. The fixture launches QEMU
before the test body and shuts it down after, so setup problems (missing
binary, QEMU not starting) are reported as pytest errors and only console
checks are reported as test failures.

--binaries-dir gives the directory holding the binaries to boot, and is
mandatory. --log-dir gives where to keep the QEMU and console logs, one
<module>/<test> subdirectory per test. The timeout of each expected string
is 1500 seconds, as in console.exp, overridden by TEST_TIMEOUT_OVERRIDE or
TEST_TIMEOUT.

Suggested-by: Stefano Stabellini <sstabellini@xxxxxxxxxx>
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Baptiste Le Duc <baptiste.le-duc@xxxxxxxxxx>
---
Changes in v3:
- New patch, split out of v2 "automation/qtb: add Python QTB framework
  with the console-test type".
- Rework on top of pytest: tests are pytest functions parametrizing the
  `machine` fixture. The CLI, config.yaml, the per-type YAML files, the
  test class hierarchy and the attempts/retry logic are gone.
- Take the binaries and logs directories from --binaries-dir and
  --log-dir, no environment variable.
- Take the per-string timeout from TEST_TIMEOUT_OVERRIDE/TEST_TIMEOUT,
  like console.exp.
---
 automation/scripts/qtb/riscv/tests/__init__.py     |  2 +
 automation/scripts/qtb/riscv/tests/conftest.py     | 65 ++++++++++++++++++++++
 automation/scripts/qtb/riscv/tests/test_console.py | 35 ++++++++++++
 3 files changed, 102 insertions(+)

diff --git a/automation/scripts/qtb/riscv/tests/__init__.py 
b/automation/scripts/qtb/riscv/tests/__init__.py
new file mode 100644
index 0000000000..24d34ae2d1
--- /dev/null
+++ b/automation/scripts/qtb/riscv/tests/__init__.py
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0-only
+"""qtb riscv64 tests: boot Xen under QEMU and check what it does."""
diff --git a/automation/scripts/qtb/riscv/tests/conftest.py 
b/automation/scripts/qtb/riscv/tests/conftest.py
new file mode 100644
index 0000000000..be6c7223a3
--- /dev/null
+++ b/automation/scripts/qtb/riscv/tests/conftest.py
@@ -0,0 +1,65 @@
+# SPDX-License-Identifier: GPL-2.0-only
+"""Fixtures booting Xen under QEMU for the qtb riscv64 tests.
+
+A test takes the `machine` fixture and parametrizes it indirectly with its
+pCPU count.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+from qtb.env import QtbEnv
+from qtb.riscv.machine import HostConfig, RiscvTestMachine
+
+def pytest_addoption(parser: pytest.Parser) -> None:
+    parser.addoption(
+        "--log-dir",
+        default=None,
+        metavar="DIR",
+        help="Directory for all logs (QEMU process log, qtest and the "
+        "consoles as con<N>.log), one subdirectory per test. When unset, "
+        "no logs are written.",
+    )
+
+    parser.addoption(
+        "--binaries-dir",
+        default=None,
+        metavar="DIR",
+        help="Directory holding the build artifacts (xen, ...) the tests 
boot.",
+    )
+
+
+@pytest.fixture
+def machine(request: pytest.FixtureRequest, qtb_env: QtbEnv):
+    """The test's machine: QEMU launched."""
+    if not hasattr(request, "param"):
+        pytest.fail("machine fixture needs indirect parametrization with the "
+                    "pCPU count")
+    machine = RiscvTestMachine(
+        HostConfig(request.param),
+        env=qtb_env,
+    )
+    with machine:
+        machine.launch()
+        yield machine
+
+
+@pytest.fixture
+def qtb_env(request: pytest.FixtureRequest) -> QtbEnv:
+    log_dir = request.config.getoption("log_dir")
+    if log_dir:
+        # One <module>/<test> subdirectory per test
+        log_dir = Path(log_dir) / request.node.path.stem / request.node.name
+        log_dir.mkdir(parents=True, exist_ok=True)
+
+    binaries_dir = request.config.getoption("binaries_dir")
+    if binaries_dir is None:
+        pytest.fail("no binaries dir: pass --binaries-dir")
+
+    return QtbEnv(
+        binaries_dir=Path(binaries_dir),
+        log_dir=log_dir,
+    )
diff --git a/automation/scripts/qtb/riscv/tests/test_console.py 
b/automation/scripts/qtb/riscv/tests/test_console.py
new file mode 100644
index 0000000000..10169e03a8
--- /dev/null
+++ b/automation/scripts/qtb/riscv/tests/test_console.py
@@ -0,0 +1,35 @@
+# SPDX-License-Identifier: GPL-2.0-only
+"""Console string expectation tests (console-test).
+
+Boot Xen under QEMU and check its console, within the timeout.
+"""
+
+from __future__ import annotations
+
+import os
+
+import pytest
+from qemu.qtb import GuestConsoleSession
+
+from qtb.riscv.machine import RiscvTestMachine
+
+# Per expected string (seconds)
+TIMEOUT: int = int(
+    os.environ.get("TEST_TIMEOUT_OVERRIDE") or os.environ.get("TEST_TIMEOUT")
+    or 1500
+)
+
+
+def _check_console(console: GuestConsoleSession, expect: list[str]) -> None:
+    """Assert each string in `expect` is printed on console, in order."""
+    for s in expect:
+        RiscvTestMachine.expect_console(console, s, TIMEOUT)
+
+
+def _check_host_boot(machine: RiscvTestMachine) -> None:
+    _check_console(machine.get_host_console(), ["All set up"])
+
+
+@pytest.mark.parametrize("machine", [pytest.param(1, id="1pcpu")], 
indirect=True)
+def test_host_boot(machine):
+    _check_host_boot(machine)

-- 
2.56.0