Re: [PATCH v2 0/3] Yocto Gitlab CI

[Michal Orzel <michal.orzel@xxxxxxx>]

Hi Bertrand,

On 19/10/2022 12:40, Bertrand Marquis wrote:
> 
> 
> Hi Michal,
> 
>> On 19 Oct 2022, at 10:06, Michal Orzel <michal.orzel@xxxxxxx> wrote:
>>
>> Hi Stefano,
>>
>> On 19/10/2022 02:02, Stefano Stabellini wrote:
>>>
>>>
>>> On Mon, 17 Oct 2022, Stefano Stabellini wrote:
>>>> It should be
>>>>
>>>> BB_NUMBER_THREADS="2"
>>>>
>>>> but that worked! Let me a couple of more tests.
>>>
>>> I could run successfully a Yocto build test with qemuarm64 as target in
>>> gitlab-ci, hurray! No size issues, no build time issues, everything was
>>> fine. See:
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fxen-project%2Fpeople%2Fsstabellini%2Fxen%2F-%2Fjobs%2F3193051236&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7C5f7fc3a161fe44b5954808dab1be5c3a%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638017728406088513%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=2mb3N26wiz39RJNSA4KoIOt%2BG9X7EMDOWIpfKc2ZZOc%3D&amp;reserved=0
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fxen-project%2Fpeople%2Fsstabellini%2Fxen%2F-%2Fjobs%2F3193083119&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7C5f7fc3a161fe44b5954808dab1be5c3a%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638017728406088513%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=QhTFefS8NU1f7oLemB0Vtn%2BDCD%2BCnq1v1gEmlKCJt84%3D&amp;reserved=0
>>>
>>> I made the appended changes in top of this series.
>>>
>>> - I pushed registry.gitlab.com/xen-project/xen/yocto:kirkstone and
>>>  registry.gitlab.com/xen-project/xen/yocto:kirkstone-qemuarm64
>>> - for the gitlab-ci runs, we need to run build-yocto.sh from the copy in
>>>  xen.git, not from a copy stored inside a container
>>> - when building the kirkstone-qemuarm64 container the first time
>>>  (outside of gitlab-ci) I used COPY and took the script from the local
>>>  xen.git tree
>>> - after a number of tests, I settled on: BB_NUMBER_THREADS="8" more than
>>>  this and it breaks on some workstations, please add it
>>> - I am running the yocto build on arm64 so that we can use the arm64
>>>  hardware to do it in gitlab-ci
>>>
>>> Please feel free to incorporate these changes in your series, and add
>>> corresponding changes for the qemuarm32 and qemux86 targets.
>>>
>>> I am looking forward to it! Almost there!
>>>
>>> Cheers,
>>>
>>> Stefano
>>>
>>>
>>> diff --git a/automation/build/yocto/build-yocto.sh 
>>> b/automation/build/yocto/build-yocto.sh
>>> index 0d31dad607..16f1dcc0a5 100755
>>> --- a/automation/build/yocto/build-yocto.sh
>>> +++ b/automation/build/yocto/build-yocto.sh
>>> @@ -107,6 +107,9 @@ IMAGE_INSTALL:append:pn-xen-image-minimal = " 
>>> ssh-pregen-hostkeys"
>>> # Save some disk space
>>> INHERIT += "rm_work"
>>>
>>> +# Reduce number of jobs
>>> +BB_NUMBER_THREADS="8"
>>> +
>>> EOF
>>>
>>>     if [ "${do_localsrc}" = "y" ]; then
>>> diff --git a/automation/build/yocto/kirkstone-qemuarm64.dockerfile 
>>> b/automation/build/yocto/kirkstone-qemuarm64.dockerfile
>>> index f279a7af92..aea3fc1f3e 100644
>>> --- a/automation/build/yocto/kirkstone-qemuarm64.dockerfile
>>> +++ b/automation/build/yocto/kirkstone-qemuarm64.dockerfile
>>> @@ -16,7 +16,8 @@ ARG target=qemuarm64
>>>
>>> # This step can take one to several hours depending on your download 
>>> bandwith
>>> # and the speed of your computer
>>> -RUN /home/$USER_NAME/bin/build-yocto.sh --dump-log $target
>>> +COPY ./build-yocto.sh /
>>> +RUN /build-yocto.sh --dump-log $target
>>>
>>> FROM $from_image
>>>
>>> diff --git a/automation/build/yocto/kirkstone.dockerfile 
>>> b/automation/build/yocto/kirkstone.dockerfile
>>> index 367a7863b6..ffbd91aa90 100644
>>> --- a/automation/build/yocto/kirkstone.dockerfile
>>> +++ b/automation/build/yocto/kirkstone.dockerfile
>>> @@ -84,9 +84,6 @@ RUN mkdir -p /home/$USER_NAME/yocto-layers \
>>>              /home/$USER_NAME/xen && \
>>>     chown $USER_NAME.$USER_NAME /home/$USER_NAME/*
>>>
>>> -# Copy the build script
>>> -COPY build-yocto.sh /home/$USER_NAME/bin/
>>> -
>>> # clone yocto repositories we need
>>> ARG yocto_version="kirkstone"
>>> RUN for rep in \
>>> diff --git a/automation/gitlab-ci/build.yaml 
>>> b/automation/gitlab-ci/build.yaml
>>> index ddc2234faf..4b8bcde252 100644
>>> --- a/automation/gitlab-ci/build.yaml
>>> +++ b/automation/gitlab-ci/build.yaml
>>> @@ -584,6 +584,22 @@ alpine-3.12-gcc-arm64-boot-cpupools:
>>>     EXTRA_XEN_CONFIG: |
>>>       CONFIG_BOOT_TIME_CPUPOOLS=y
>>>
>>> +yocto-kirkstone-qemuarm64:
>>> +  stage: build
>>> +  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
>>> +  script:
>>> +    - ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs 
>>> --xen-dir=`pwd` qemuarm64
>>> +  variables:
>>> +    CONTAINER: yocto:kirkstone-qemuarm64
>>> +  artifacts:
>>> +    paths:
>>> +      - '*.log'
>>> +      - '*/*.log'
>> The above lines are not needed as the logs/* below will handle them all 
>> (logs are only stored in logs/).
> 
> Ack
> 
>>
>>> +      - 'logs/*'
>>> +    when: always
>>> +  tags:
>>> +    - arm64
>>> +
>> build-yocto.sh performs both build and run actions. I think it'd be better 
>> to move this into test.yaml in that case.
>> The best would be to create one build job (specifying --no-run) in 
>> build.yaml and one test job (specifying --no-build) in test.yaml.
>> This however would probably require marking path 
>> build/tmp/deploy/***/qemuarm64 as an build artifact. The question then is
>> whether having this path would be enough for runqemu (Bertrand's opinion 
>> needed).
> 
> This will not be enough to run qemu as the qemu binary and its dependencies 
> are in the build artifacts and not in deploy.
> Splitting the build and run is not a good idea because the size of the 
> artifact between the 2 will be huge.
> 
>>
>> Apart from that there is an aspect of Yocto releases and the 
>> containers/tests names.
>> Yocto needs to be up-to-date in order to properly build Xen+tools.
>> This basically means that we will need to update the containers once
>> per Yocto release. The old containers would still need to be stored in our 
>> CI container registry
>> so that we can use CI for older versions of Xen. However, updating the 
>> containers would also require
>> modifying the existing tests (for now we have e.g. yocto-kirkstone-qemuarm64 
>> but in a month we will have
>> to change them to yocto-langdale-qemuarm64). In a few years time this will 
>> result in several CI jobs
>> that are the same but differ only in name/container. I would thus suggest to 
>> name the CI jobs like this:
>> yocto-qemuarm64 (without yocto release name) and define the top-level 
>> YOCTO_CONTAINER variable to store
>> the current yocto release container. This will solve the issue I described 
>> above.
> 
> I think we have no other way around this and we will need to have one Yocto 
> release supported by Xen officially so
> we will have to keep old docker images for old releases of Xen and move to 
> newer versions of Yocto in staging when
> it is needed.
> 
> We have to find a way for gitlab-ci to use the build.yaml contained inside 
> the tree that is to be tested somehow so that gitlab would automatically take 
> the right one.
> Which means that build.yaml will be different between branches and contain 
> the right version for the current branch.
> 

What I suggest is that with each new yocto release, we add new docker container 
files and push them to registry.
So we will end up in a registry having e.g. (arm64 as an example):
- kirkstone-qemuarm64
- langdale-qemuarm64
We maintain only the one group of CI jobs whose names are generic 
(yocto-qemuarm64).
After adding new containers for a new Yocto release, we modify the 
YOCTO_RELEASE variable
to point to the latest yocto release containers.

test.yaml:
...
# Yocto test jobs
variables:
  YOCTO_RELEASE: "kirkstone"

yocto-qemuarm64:
  extends: .test-jobs-common
  script:
    - ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs 
--xen-dir=`pwd` qemuarm64
  variables:
    CONTAINER: yocto:${YOCTO_RELEASE}-qemuarm64
  artifacts:
    paths:
      - 'logs/*'
    when: always
  tags:
    - arm64

This means that:
- on the current staging branch the YOCTO_RELEASE points to the latest 
containers (for the latest yocto release)
- on the old stable branches the YOCTO_RELEASE points to the old containers 
(for the old yocto release).

~Michal