Re: [PATCH v1 3/3] automation: allow selecting individual jobs via CI variables

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

On Thu, Feb 13, 2025 at 05:36:47PM -0800, Stefano Stabellini wrote:
> On Thu, 13 Feb 2025, Marek Marczykowski-Górecki wrote:
> > Debugging sometimes involves running specific jobs on different
> > versions. It's useful to easily avoid running all of the not interesting
> > ones (for given case) to save both time and CI resources. Doing so used
> > to require changing the yaml files, usually in several places.
> > Ease this step by adding SELECTED_JOBS_ONLY variable that takes a regex.
> > Note that one needs to satisfy job dependencies on their own (for
> > example if a test job needs a build job, that specific build job
> > needs to be included too).
> > 
> > The variable can be specified via Gitlab web UI when scheduling a
> > pipeline, but it can be also set when doing git push directly:
> > 
> >     git push -o ci.variable=SELECTED_JOBS_ONLY="/job1|job2/"
> > 
> > More details at https://docs.gitlab.co.jp/ee/user/project/push_options.html
> > 
> > The variable needs to include regex for selecting jobs, including
> > enclosing slashes.
> > A coma/space separated list of jobs to select would be friendlier UX,
> > but unfortunately that is not supported:
> > https://gitlab.com/gitlab-org/gitlab/-/issues/209904 (note the proposed
> > workaround doesn't work for job-level CI_JOB_NAME).
> > On the other hand, the regex is more flexible (one can select for
> > example all arm32 jobs).
> 
> I was trying to find workarounds so that we could also support the
> simple list of comma-separated jobs you mentioned, but I couldn't find
> an easy way to do that.
> 
> However, one thing we can do is to support writing SELECTED_JOBS_ONLY in
> .gitlab-ci.yml as a commit in xen.git, for people that don't know or
> don't remember how to set ci.variable using the git command line.

You can always do it, in `variables` setting AFAIR.

> Given that this is for testing, I think it would be no problem to adding
> a special commit on top of your tree. We are just trying to make it
> easier compared to having to manually delete the list of jobs we don't
> need.

Yes, manually delete was awful. In practice I usually added always-false
rules, but still.

> But even with the special commit, I couldn't think of an easy way to
> achieve the nicer comma-separated list of jobs...
> 
> 
> > Signed-off-by: Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>
> > ---
> > This probably wants documenting beyond this commit message. I don't
> > think we have any CI-related docs anywhere, do we? Some new file in
> > docs/misc?
> 
> Yes please :-)
> 
> It would be also worth documenting how to create a simple pipeline by
> removing everything that you don't need for a single test

You mean how to find what jobs you need?

> > And also, it's possible to extend web ui for starting pipelines to
> > include pre-defined variables. I use it in qubes here if you want to
> > see:
> > https://gitlab.com/QubesOS/qubes-continuous-integration/-/pipelines/new
> 
> I don't have access to this

Oh, sorry. Screenshot attached.

And its definition looks like this:
https://gitlab.com/QubesOS/qubes-continuous-integration/-/blob/main/.gitlab-ci.yml?ref_type=heads#L15-26

> > Does it make sense to include SELECTED_JOBS_ONLY this way too?
> > Personally, I'll probably use it via cmdline push only anyway, but I
> > don't know what workflows other people have.
> > ---
> >  automation/gitlab-ci/build.yaml |  6 ++++++
> >  automation/gitlab-ci/test.yaml  | 14 ++++++++++++++
> >  2 files changed, 20 insertions(+)
> > 
> > diff --git a/automation/gitlab-ci/build.yaml 
> > b/automation/gitlab-ci/build.yaml
> > index 35e224366f62..f12de00a164a 100644
> > --- a/automation/gitlab-ci/build.yaml
> > +++ b/automation/gitlab-ci/build.yaml
> > @@ -12,6 +12,12 @@
> >        - '*/*.log'
> >      when: always
> >    needs: []
> > +  rules:
> > +  - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> > +    when: always
> > +  - if: $SELECTED_JOBS_ONLY
> > +    when: never
> > +  - when: on_success
> >  
> >  .gcc-tmpl:
> >    variables: &gcc
> > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> > index c21a37933881..93632f1f9204 100644
> > --- a/automation/gitlab-ci/test.yaml
> > +++ b/automation/gitlab-ci/test.yaml
> > @@ -1,6 +1,11 @@
> >  .test-jobs-common:
> >    stage: test
> >    image: ${XEN_REGISTRY}/${CONTAINER}
> > +  rules:
> > +  - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> > +  - if: $SELECTED_JOBS_ONLY
> > +    when: never
> > +  - when: on_success
> >  
> >  .arm64-test-needs: &arm64-test-needs
> >    - alpine-3.18-arm64-rootfs-export
> > @@ -99,6 +104,9 @@
> >        - '*.dtb'
> >      when: always
> >    rules:
> > +    - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> > +    - if: $SELECTED_JOBS_ONLY
> > +      when: never
> >      - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
> >    tags:
> >      - xilinx
> > @@ -117,6 +125,9 @@
> >        - '*.log'
> >      when: always
> >    rules:
> > +    - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> > +    - if: $SELECTED_JOBS_ONLY
> > +      when: never
> >      - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
> >    tags:
> >      - xilinx
> > @@ -137,6 +148,9 @@
> >        - '*.log'
> >      when: always
> >    rules:
> > +    - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY
> > +    - if: $SELECTED_JOBS_ONLY
> > +      when: never
> >      - if: $QUBES_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
> >    tags:
> >      - qubes-hw2
> > -- 
> > git-series 0.9.1
> > 


-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachment: pngfeAPNTP6PX.png
Description: PNG image

Attachment: signature.asc
Description: PGP signature