Re: [RFC XEN PATCH 00/13] Running bare-metal tests with "pytest"

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

On Tue, Sep 22, 2026 at 12:08:47PM +0200, Anthony PERARD wrote:
> From: Anthony PERARD <anthony.perard@xxxxxxxxxx>
> 
> Patch series available in this git branch:
> https://xenbits.xenproject.org/git-http/people/aperard/xen-unstable.git 
> br.ci.bare-metal-with-pytest-v1
> 
> Hi,
> 
> I wanted to add some new machine to our GitLab CI, but I didn't want to
> duplicate yet another time the existing shell script. They are good to get the
> ball rolling but are getting harder to maintain as more test are been added,
> and don't share common code between machines.
>
> So, I've start work on something based on `pytest` which I hope will be easier
> to maintain and extend and use to add new machines.

Nice!

> It's still Work In Progress.
> 
> Here is the current result:
>     https://gitlab.com/xen-project/people/anthonyper/xen/-/jobs/16529107619

Minor readability improvement idea: force color output of pytest.

> In this pipeline, the artifact use are a bit different from master; linux is
> built with network driver, and vlan; the rootfs is from Marek's patch series,
> with networking and dropbear service enabled.
> 
> Next:
> 
> I'd like to at least have a new fixture that give a started Host that a test
> can ssh into. Then maybe a new class "Host" which could have functions for ssh
> and getting the serial console output.
> 
> Don't go to much into detail review, I'm more interested of a general 
> overview,
> and to present what I have so far.

We had a design session on this topic, and we discussed a structural
change: have one job per Xen boot flavor (mostly dom0 PV vs PVH, but
could be also other settings needing host reboot) and then have pytest
run several tests via SSH. The idea is to not waste 2-4 minutes for
every ~10 sec test. Yes, it does mean one failed test will likely
prevent others in the same job from running (especially in case of Xen
panic), but since the intended state is "all green", a single failure is
bad already, and this approach should still get you enough information
to debug that first failure.

In practical terms, it would require most fixtures to be class or
session scope, instead of the default test scope. And using more SSH
instead of baking tests into startup scripts (already partially done,
but may need extending to domU commands too).

Other than that, this approach looks nice, and I really like using
pytest here, as it makes the structure significantly better. This will
likely require a bit of adjustments to handle also other hosts (for
example for qubes runners tftp dir is mounted into gitlab's container so
there is no need to sftp boot files). But that should be easy to do
later, it doesn't change the overall structure much.


> Some runes:
>     - uv run mypy .
>       to check the code, at least the types used.
>       other code check could be added, and run by the pipeline
>     - uv run pytest
>       to run the whole test suite
>     - uv run pytest test_boot.py::test_xen_tools_tests
>       to run just one test
> 
> I did create an Arch Linux docker image with pytest, but paramiko got too new
> to be used, so I've started to use `uv` to deal with the dependencies. `mypy`
> always needed to be run with `uv` due to missing types for paramiko (not
> packaged in Arch Linux). So the image is useless, and I'll look for a 
> different
> image later, for now just Arch Linux with `uv` installed.

FWIW Qubes tests use Alpine container, seems to have all what we need. I
see it also has paramiko 4.0.0

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachment: signature.asc
Description: PGP signature