Re: [PATCH v2 3/5] automation: Add the expect script with test case for FVP

[Michal Orzel <michal.orzel@xxxxxxx>]

On 08/12/2023 10:05, Henry Wang wrote:
> 
> 
> Hi Michal,
> 
>> On Dec 8, 2023, at 16:57, Michal Orzel <michal.orzel@xxxxxxx> wrote:
>>
>> Hi Henry,
>>
>> On 08/12/2023 06:46, Henry Wang wrote:
>>>
>>>
>>> To interact with the FVP (for example entering the U-Boot shell
>>> and transferring the files by TFTP), we need to connect the
>>> corresponding port by the telnet first. Use an expect script to
>>> simplify the automation of the whole "interacting with FVP" stuff.
>>>
>>> The expect script will firstly detect the IP of the host, then
>>> connect to the telnet port of the FVP, set the `serverip` and `ipaddr`
>>> for the TFTP service in the U-Boot shell, and finally boot Xen from
>>> U-Boot and wait for the expected results by Xen, Dom0 and DomU.
>>>
>>> Signed-off-by: Henry Wang <Henry.Wang@xxxxxxx>
>> Reviewed-by: Michal Orzel <michal.orzel@xxxxxxx>
> 
> Thanks!
> 
>> with 1 question...
>>
>>> ---
>>> v2:
>>> - No change.
>>> ---
>>> .../expect/fvp-base-smoke-dom0-arm64.exp      | 73 +++++++++++++++++++
>>> 1 file changed, 73 insertions(+)
>>> create mode 100755 automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
>>>
>>> diff --git a/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp 
>>> b/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
>>> new file mode 100755
>>> index 0000000000..25d9a5f81c
>>> --- /dev/null
>>> +++ b/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
>>> @@ -0,0 +1,73 @@
>>> +#!/usr/bin/expect
>>> +
>>> +set timeout 2000
>> Do we really need such a big timeout (~30 min)?
>> Looking at your test job, it took 16 mins (quite a lot but I know FVP is slow
>> + send_slow slows things down)
> 
> This is a really good question. I did have the same question while working on
> the negative test today. The timeout 2000 indeed will fail the job at about 
> 30min,
> and waiting for it is indeed not really pleasant.
> 
> But my second thought would be - from my observation, the overall time now
> would vary between 15min ~ 20min, and having a 10min margin is not that crazy
> given that we probably will do more testing from the job in the future, and 
> if the
> GitLab Arm worker is high loaded, FVP will probably become slower. And 
> normally
> we don’t even trigger the timeout as the job will normally pass. So I decided
> to keep this.
> 
> Mind sharing your thoughts about the better value of the timeout? Probably 
> 25min?
>From what you said that the average is 15-20, I think we can leave it set to 
>30.
But I wonder if we can do something to decrease the average time. ~20 min is a 
lot
even for FVP :) Have you tried setting send_slow to something lower than 100ms?
That said, we don't send too many chars to FVP, so I doubt it would play a 
major role
in the overall time.

I use FVP quite rarely these days, so you should know better if this can be 
perceived as
usual/normal behavior.

~Michal