Re: [PATCH v5 1/3] automation: Create Yocto docker images

[Michal Orzel <michal.orzel@xxxxxxx>]

Hi Bertrand,

I will try to build and run the yocto containers for a test.
In the meantime I just spotted one thing.

On 30/11/2022 13:15, Bertrand Marquis wrote:
> 
> 
> Add containers suitable to run yocto kirkstone build based on ubuntu
> 22.04. It contains all packages required by Yocto and a checkout of the
> layers required to build Xen with Yocto.
> 
> Add a generic docker image template to be used to automatically generate
> docker files for different configurations:
> - specific yocto version
> - different targets (qemu arm, arm64 and x86)
> - different host platforms (x86 or arm64)
> 
> During a call to 'make all', only the images for the host platform we
> run it on in the CI is generated.
> If needed, images for an other host platform can be generated manually
> by calling the right make target (see make help).
> 
> Add a build script to build and run xen on qemu using Yocto.
> The script supports arm32, arm64 and x86_64 and checks that dom0 is
> properly booting. At this stage this does not run any guest on top of
> dom0. The script is to be executed in one of the docker images to build
> and run a system using a Xen source tree.
> 
> Signed-off-by: Bertrand Marquis <bertrand.marquis@xxxxxxx>
> ---
> Changes in v5:
> - typo fixes
> - add INTERMEDIATE to remove intermediate dockerfile
> - use container architecture
> - use full words in foreach loops
> - fix shellcheck findings in build-yocto.sh
> - rebase on sstabellini next branch
> Changes in v4:
> - Rework the system to have one dockerfile template from which make will
> generate the required dockerfiles for the wanted configuration
> - add support for different host architectures
> - Merge the generation system into one single dockerfile
> - Merge patches 1 and 2 in a single patch
> - Introduce CONTAINERS_EXTRA to have extra containers not built by
> default (for those not used by CI but useful to users)
> Changes in v3:
> - limit number of jobs in yocto by default to 8 and add --num-jobs
> option to the script to set a custom number of jobs
> - do not copy anymore the build-yocto.sh script inside the main image so
> that the current one in the repository is used when running
> Changes in v2:
> - add a --dump-log command line option to build-yocto.sh script to dump
> the logs if an error occurs.
> Changes in v1:
> - add --image command line argument to build-yocto.sh to allow building
> something different than xen-image-minimal.
> - modify dockerfile to have one layer per line and make it easier to add
> other. I kept the for loop to keep the number of docker steps lower
> - update commit message to warn that no guest are tested.
> - fix build-yocto script to properly return with an error if expect
> script ends up in timeout or EOF.
> ---
>  automation/build/Makefile                  |  14 +-
>  automation/build/yocto/build-yocto.sh      | 351 +++++++++++++++++++++
>  automation/build/yocto/yocto.dockerfile.in | 114 +++++++
>  automation/build/yocto/yocto.inc           |  42 +++
>  4 files changed, 519 insertions(+), 2 deletions(-)
>  create mode 100755 automation/build/yocto/build-yocto.sh
>  create mode 100644 automation/build/yocto/yocto.dockerfile.in
>  create mode 100644 automation/build/yocto/yocto.inc
> 
> diff --git a/automation/build/Makefile b/automation/build/Makefile
> index a4b2b85178cf..72a5335baec1 100644
> --- a/automation/build/Makefile
> +++ b/automation/build/Makefile
> @@ -1,13 +1,18 @@
> 
>  # the base of where these containers will appear
>  REGISTRY := registry.gitlab.com/xen-project/xen
> -CONTAINERS = $(subst .dockerfile,,$(wildcard */*.dockerfile))
> +CONTAINERS = $(filter-out yocto/%,$(subst .dockerfile,,$(wildcard 
> */*.dockerfile)))
> +CONTAINERS_EXTRA =
>  DOCKER_CMD ?= docker
> 
> +include yocto/yocto.inc
Because this appears before help, when calling make without any target like:
make -C automation/build
the previous behavior of printing help by default is no longer true (the 
default target is
taken from yocto.inc in this case). Instead we're getting:

make: Entering directory...
rm yocto/kirkstone-qemuarm64.dockerfile
make: Leaving directory...

This include can be easily moved after help to stick to the current behavior 
and still
achieve what we want (i.e. make will still first evaluate include and then call 
the first
target which is help, so CONTAINERS and CONTAINERS_EXTRA will contain correct 
data).

> +
>  help:
>         @echo "Builds containers for building Xen based on different distros"
>         @echo "To build one run 'make DISTRO/VERSION'. Available containers:"
>         @$(foreach file,$(sort $(CONTAINERS)),echo ${file};)
> +       @echo "Extra containers (not built using make all):"
> +       @$(foreach file,$(sort $(CONTAINERS_EXTRA)),echo ${file};)
>         @echo "To push container builds, set the env var PUSH"
> 
>  %: %.dockerfile ## Builds containers
> @@ -16,5 +21,10 @@ help:
>                 $(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
>         fi
> 
> -.PHONY: all
> +.PHONY: all clean
>  all: $(CONTAINERS)
> +
> +# Remove generated dockerfiles for yocto
> +clean:
> +       rm -f yocto/*.dockerfiles
> +

~Michal