[RFC XEN PATCH 00/13] Running bare-metal tests with "pytest"

[Anthony PERARD <anthony@xxxxxxxxxxxxxx>]

From: Anthony PERARD <anthony.perard@xxxxxxxxxx>

Patch series available in this git branch:
https://xenbits.xenproject.org/git-http/people/aperard/xen-unstable.git 
br.ci.bare-metal-with-pytest-v1

Hi,

I wanted to add some new machine to our GitLab CI, but I didn't want to
duplicate yet another time the existing shell script. They are good to get the
ball rolling but are getting harder to maintain as more test are been added,
and don't share common code between machines.

So, I've start work on something based on `pytest` which I hope will be easier
to maintain and extend and use to add new machines.

It's still Work In Progress.

Here is the current result:
    https://gitlab.com/xen-project/people/anthonyper/xen/-/jobs/16529107619

In this pipeline, the artifact use are a bit different from master; linux is
built with network driver, and vlan; the rootfs is from Marek's patch series,
with networking and dropbear service enabled.

Next:

I'd like to at least have a new fixture that give a started Host that a test
can ssh into. Then maybe a new class "Host" which could have functions for ssh
and getting the serial console output.

Don't go to much into detail review, I'm more interested of a general overview,
and to present what I have so far.

Some runes:
    - uv run mypy .
      to check the code, at least the types used.
      other code check could be added, and run by the pipeline
    - uv run pytest
      to run the whole test suite
    - uv run pytest test_boot.py::test_xen_tools_tests
      to run just one test

I did create an Arch Linux docker image with pytest, but paramiko got too new
to be used, so I've started to use `uv` to deal with the dependencies. `mypy`
always needed to be run with `uv` due to missing types for paramiko (not
packaged in Arch Linux). So the image is useless, and I'll look for a different
image later, for now just Arch Linux with `uv` installed.

Thanks,

Anthony PERARD (13):
  pytest: Add fixture to prepare rootfs
  Creating new python project
  pytest: Add fixture to prepare the tftp server
  pytest: Add fixture that boot a machine
  pytest: Introduce reading the console, and add a test
  We need an older version of paramiko.
  pytest: Add random string to look for in the logs
  pytest: Run run-tools-tests over ssh on the test machine
  pytest: Generate a rootfs for a guest
  pytest: Add guest test
  pytest: Add config to enable live log, and disable ssh/paramiko
    logging
  pytest: Add example .env for options for the tests
  gitlab-ci: Add job for moonshot with pytest

 automation/gitlab-ci/test.yaml       |  21 +++
 automation/pytest/.env               |  19 ++
 automation/pytest/conftest.py        |   6 +
 automation/pytest/lib/__init__.py    |   0
 automation/pytest/lib/boot_binary.py | 261 +++++++++++++++++++++++++++
 automation/pytest/lib/ipmi.py        | 155 ++++++++++++++++
 automation/pytest/lib/net_boot.py    |  60 ++++++
 automation/pytest/pyproject.toml     |  17 ++
 automation/pytest/pytest.toml        |  13 ++
 automation/pytest/test_boot.py       | 237 ++++++++++++++++++++++++
 10 files changed, 789 insertions(+)
 create mode 100644 automation/pytest/.env
 create mode 100644 automation/pytest/conftest.py
 create mode 100644 automation/pytest/lib/__init__.py
 create mode 100644 automation/pytest/lib/boot_binary.py
 create mode 100644 automation/pytest/lib/ipmi.py
 create mode 100644 automation/pytest/lib/net_boot.py
 create mode 100644 automation/pytest/pyproject.toml
 create mode 100644 automation/pytest/pytest.toml
 create mode 100644 automation/pytest/test_boot.py

-- 
Anthony PERARD