[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH v1] automation: fix false success in qemu tests
- To: xen-devel@xxxxxxxxxxxxxxxxxxxx
- From: "Lira, Victor M" <VictorM.Lira@xxxxxxx>
- Date: Wed, 28 Aug 2024 21:18:12 -0700
- Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none
- Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CuIZrbkzZ1dOSGUoyKupye1CIvOEvqvYkiVLKnr3yLA=; b=VwgNa56HuZepH8OHlA3oX3xJb7o6NlUVBHmTSeCPvFrqOIi0EGvIpX7al/zhzeOK8neDRkmxI16InOa1vIeVg3AvLqJ+j3TnAVcZN0Mlo2Y0ddkK1FCclbY7r7PZMsEKN3mjt28zCXBvlI/8+El0Nh/fpHFHQJuom87Ocjr5zgUYCwQVwAahkHS3X1+jfWA1cN72AKBA6hDw8gi2/EM3XpQ/B9CqqnJaNzLgpQd0Y/YIseVVR4nTiI+Eeq714ll3siLPtHBSYWogyseZ8c4RTMsNmuLcMYk9laMWCoVi1QtbxBPdlALEjBPQtYFqczKSUp6lQ02Qutiq9tS2KXu5NA==
- Arc-seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=QpX87+616XPT/uw3/Wa2fcjqSIicKhi6AAs/a/vNLhmFJbwpoDZYoBL/iYxXwM9mx4GnHGWSJz5OXM3wRQjMUf+0YxU75MCeaDYvVKrDqA/PLAk1eZlrvLKrUKJ3tCktQ3MeVE5V3X/QAEOj8ci+nMT/og0AQfcjchp25P5Yu6OB89L4RrLCDWVd8BUkYNkQ2n/FnHjX2DH9tRaeqWoAlnW1RZOvEnW9eau5PorNVNuUseioYGLwlKVxfj+UOSg67vN/ZZ0r67+AKYqXLwdeSdOMTTxhkp96Y031NjbR+aG8Y9KQq8enkq6pvn39DyRPFwSGr0LLjVf6ZhJKW7TbDQ==
- Authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com;
- Cc: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>, Jan Beulich <jbeulich@xxxxxxxx>, Julien Grall <julien@xxxxxxx>, Stefano Stabellini <sstabellini@xxxxxxxxxx>
- Delivery-date: Thu, 29 Aug 2024 04:18:23 +0000
- List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
While developing a patch to update xilinx tests to use the "expect"
utility I found that with the current use of "expect" in the *qemu-*
*scripts,
it is possible for the test to produce a *false success *if the expect
script returns nonzero due to the following lines
set +e
...
./automation/scripts/qemu-key.exp | sed 's/\r\+$//'
(end of file)
The first disables exiting on errors which is needed for cleanup in some
cases such as turning off power in a hardware test. However, here the
script exits 0 on reaching the end of file.
This is a serious flaw in our testing system as the result of the test is
essentially ignored.
I have been able produce such a false success here by setting a random
passed
message and short timeout (line 142).
https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/7693194287#L142
I have sent a patch that fixes this using the same method from my xilinx
patch but suggestions are welcome. For qemu tests it may be enough to
instead just remove "set +e".
|
