Re: [PATCH v2 3/5] automation: Add the expect script with test case for FVP

[Henry Wang <Henry.Wang@xxxxxxx>]

Hi Wei, Michal,

> On Dec 8, 2023, at 19:13, Wei Chen <Wei.Chen@xxxxxxx> wrote:
> 
> Hi Henry and Michal,
> 
>>>>>>> On 08/12/2023 06:46, Henry Wang wrote:
>>>>>>>> diff --git a/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp 
>>>>>>>> b/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
>>>>>>>> new file mode 100755
>>>>>>>> index 0000000000..25d9a5f81c
>>>>>>>> --- /dev/null
>>>>>>>> +++ b/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
>>>>>>>> @@ -0,0 +1,73 @@
>>>>>>>> +#!/usr/bin/expect
>>>>>>>> +
>>>>>>>> +set timeout 2000
>>>>>>> Do we really need such a big timeout (~30 min)?
>>>>>>> Looking at your test job, it took 16 mins (quite a lot but I know FVP 
>>>>>>> is slow
>>>>>>> + send_slow slows things down)
>>>>>> 
>>>>>> This is a really good question. I did have the same question while 
>>>>>> working on
>>>>>> the negative test today. The timeout 2000 indeed will fail the job at 
>>>>>> about 30min,
>>>>>> and waiting for it is indeed not really pleasant.
>>>>>> 
>>>>>> But my second thought would be - from my observation, the overall time 
>>>>>> now
>>>>>> would vary between 15min ~ 20min, and having a 10min margin is not that 
>>>>>> crazy
>>>>>> given that we probably will do more testing from the job in the future, 
>>>>>> and if the
>>>>>> GitLab Arm worker is high loaded, FVP will probably become slower. And 
>>>>>> normally
>>>>>> we don’t even trigger the timeout as the job will normally pass. So I 
>>>>>> decided
>>>>>> to keep this.
>>>>>> 
>>>>>> Mind sharing your thoughts about the better value of the timeout? 
>>>>>> Probably 25min?
>>>>> From what you said that the average is 15-20, I think we can leave it set 
>>>>> to 30.
>>>>> But I wonder if we can do something to decrease the average time. ~20 min 
>>>>> is a lot
>>>>> even for FVP :) Have you tried setting send_slow to something lower than 
>>>>> 100ms?
>>>>> That said, we don't send too many chars to FVP, so I doubt it would play 
>>>>> a major role
>>>>> in the overall time.
>>>> 
>>>> I agree with the send_slow part. Actually I do have the same concern, here 
>>>> are my current
>>>> understanding and I think you will definitely help with your knowledge:
>>>> If you check the full log of Dom0 booting, for example [1], you will find 
>>>> that we wasted so
>>>> much time in starting the services of the OS (modloop, udev-settle, etc). 
>>>> All of these services
>>>> are retried many times but in the end they are still not up, and from my 
>>>> understanding they
>>>> won’t affect the actual test(?) If we can somehow get rid of these 
>>>> services from rootfs, I think
>>>> we can save a lot of time.
>>>> 
>>>> And honestly, I noticed that qemu-alpine-arm64-gcc suffers from the same 
>>>> problem and it also
>>>> takes around 15min to finish. So if we managed to tailor the services from 
>>>> the filesystem, we
>>>> can save a lot of time.
>>> That is not true. Qemu runs the tests relatively fast within few minutes. 
>>> The reason you see e.g. 12 mins
>>> for some Qemu jobs comes from the timeout we set in Qemu scripts. We don't 
>>> have yet the solution (we could
>>> do the same as Qubes script) to detect the test success early and exit 
>>> before timeout. That is why currently
>>> the only way for Qemu tests to finish is by reaching the timeout.
>>> 
>>> So the problem is not with the rootfs and services (the improvement would 
>>> not be significant) but with
>>> the simulation being slow. That said, this is something we all know and I 
>>> expect FVP to only be used in scenarios
>>> which cannot be tested using Qemu or real HW.
>> Ok, you made a point. Let me do some experiments to see if I can improve. 
>> Otherwise maybe
>> we can live with this until a better solution.
>> Kind regards,
>> Henry
> 
> QEMU works like FVP enabled use_real_time flag. How about enable 
> use_real_time flag in CI for most test cases, but disable it for
> some time sensitive test cases? Normally, enable use_real_time
> will give several times improvement of FVP performance.

I am seeing below from the FVP parameter lists of the one we are currently 
using. The "Deprecated" word worries
me a bit (the old version FVP does not have the “Deprecated" though).
```
bp.refcounter.use_real_time=0                         # (bool  , init-time) 
default = '0'      : **Deprecated, this parameter will be removed in future 
versions** Update the Generic Timer counter at a real-time base frequency 
instead of simulator time
```

Also, from my testing in the GitLab pipeline, I was not able to see significant 
time improvement. So I guess
instead I will try what Julien suggests to see if things can be better.

Kind regards,
Henry 

> 
> Cheers,
> Wei Chen
> 
>>> 
>>> ~Michal