Re: [PATCH v2 3/5] automation: Add the expect script with test case for FVP

[Henry Wang <Henry.Wang@xxxxxxx>]

Hi Michal,

> On Dec 8, 2023, at 17:11, Michal Orzel <michal.orzel@xxxxxxx> wrote:
> On 08/12/2023 10:05, Henry Wang wrote:
>> 
>> Hi Michal,
>> 
>>> On Dec 8, 2023, at 16:57, Michal Orzel <michal.orzel@xxxxxxx> wrote:
>>> 
>>> Hi Henry,
>>> 
>>> On 08/12/2023 06:46, Henry Wang wrote:
>>>> diff --git a/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp 
>>>> b/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
>>>> new file mode 100755
>>>> index 0000000000..25d9a5f81c
>>>> --- /dev/null
>>>> +++ b/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
>>>> @@ -0,0 +1,73 @@
>>>> +#!/usr/bin/expect
>>>> +
>>>> +set timeout 2000
>>> Do we really need such a big timeout (~30 min)?
>>> Looking at your test job, it took 16 mins (quite a lot but I know FVP is 
>>> slow
>>> + send_slow slows things down)
>> 
>> This is a really good question. I did have the same question while working on
>> the negative test today. The timeout 2000 indeed will fail the job at about 
>> 30min,
>> and waiting for it is indeed not really pleasant.
>> 
>> But my second thought would be - from my observation, the overall time now
>> would vary between 15min ~ 20min, and having a 10min margin is not that crazy
>> given that we probably will do more testing from the job in the future, and 
>> if the
>> GitLab Arm worker is high loaded, FVP will probably become slower. And 
>> normally
>> we don’t even trigger the timeout as the job will normally pass. So I decided
>> to keep this.
>> 
>> Mind sharing your thoughts about the better value of the timeout? Probably 
>> 25min?
> From what you said that the average is 15-20, I think we can leave it set to 
> 30.
> But I wonder if we can do something to decrease the average time. ~20 min is 
> a lot
> even for FVP :) Have you tried setting send_slow to something lower than 
> 100ms?
> That said, we don't send too many chars to FVP, so I doubt it would play a 
> major role
> in the overall time.

I agree with the send_slow part. Actually I do have the same concern, here are 
my current
understanding and I think you will definitely help with your knowledge:
If you check the full log of Dom0 booting, for example [1], you will find that 
we wasted so
much time in starting the services of the OS (modloop, udev-settle, etc). All 
of these services
are retried many times but in the end they are still not up, and from my 
understanding they
won’t affect the actual test(?) If we can somehow get rid of these services 
from rootfs, I think
we can save a lot of time.

And honestly, I noticed that qemu-alpine-arm64-gcc suffers from the same 
problem and it also
takes around 15min to finish. So if we managed to tailor the services from the 
filesystem, we
can save a lot of time.

But I found it difficult to do the proper tailoring, any suggestions?

[1] 
https://gitlab.com/xen-project/people/henryw/xen/-/jobs/5708557850/artifacts/file/smoke.serial

Kind regards,
Henry

> I use FVP quite rarely these days, so you should know better if this can be 
> perceived as
> usual/normal behavior.
> 
> ~Michal
>