[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [PATCH v2 3/4] automation: allow selecting individual jobs via CI variables
On Fri, Feb 14, 2025 at 04:29:17PM -0800, Stefano Stabellini wrote: > On Fri, 14 Feb 2025, Marek Marczykowski-Górecki wrote: > > Debugging sometimes involves running specific jobs on different > > versions. It's useful to easily avoid running all of the not interesting > > ones (for given case) to save both time and CI resources. Doing so used > > to require changing the yaml files, usually in several places. > > Ease this step by adding SELECTED_JOBS_ONLY variable that takes a regex. > > Note that one needs to satisfy job dependencies on their own (for > > example if a test job needs a build job, that specific build job > > needs to be included too). > > > > The variable can be specified via Gitlab web UI when scheduling a > > pipeline, but it can be also set when doing git push directly: > > > > git push -o ci.variable=SELECTED_JOBS_ONLY="/job1|job2/" > > > > More details at https://docs.gitlab.co.jp/ee/user/project/push_options.html > > > > The variable needs to include regex for selecting jobs, including > > enclosing slashes. > > Does it work with a single job like this? > > git push -o ci.variable=SELECTED_JOBS_ONLY="job1" No, it works with: git push -o ci.variable=SELECTED_JOBS_ONLY="/job1/" or rather: git push -o ci.variable=SELECTED_JOBS_ONLY="/^job1$/" > If it does, is there any way we could use to automatically whitelist its > dependencies too? Because that would be so much easier to use... I guess it should be possible to add some extra condition for dependencies, like extending rules for alpine-3.18-gcc-debug specifically with - if: $SELECTED_JOBS_ONLY && "adl-smoke-x86-64-gcc-debug" =~ $SELECTED_JOBS_ONLY when: always (and repeated for other tests depending on this build job) But that means dependencies need to be kept in sync manually, in two places. The absolute lack of any variables processing (even a simple string concatenation...) at this stage of gitlab yaml processing makes it challenging to propose any even semi-reasonable solution... On the other hand, if you care about specific test, you can easily get its dependencies by either looking at test.yaml, or clicking "show depdendencies" in gitlab ui and hovering over the job you want. > > A coma/space separated list of jobs to select would be friendlier UX, > > but unfortunately that is not supported: > > https://gitlab.com/gitlab-org/gitlab/-/issues/209904 (note the proposed > > workaround doesn't work for job-level CI_JOB_NAME). > > On the other hand, the regex is more flexible (one can select for > > example all arm32 jobs). > > > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx> > > --- > > This probably wants documenting beyond this commit message. I don't > > think we have any CI-related docs anywhere, do we? Some new file in > > docs/misc? > > Please add documentation for it. Other than that, I think it could be > committed. I would prefer you also added documentation about alternative > methods such as removing all the jobs except the ones you care about. Do we really want to recommend removing unnecessary jobs, given a better option exists now? I know https://xkcd.com/1171/, but still... > > And also, it's possible to extend web ui for starting pipelines to > > include pre-defined variables. I use it in qubes here if you want to > > see: > > https://gitlab.com/QubesOS/qubes-continuous-integration/-/pipelines/new > > Does it make sense to include SELECTED_JOBS_ONLY this way too? > > Personally, I'll probably use it via cmdline push only anyway, but I > > don't know what workflows other people have. > > --- > > automation/gitlab-ci/build.yaml | 6 ++++++ > > automation/gitlab-ci/test.yaml | 14 ++++++++++++++ > > 2 files changed, 20 insertions(+) > > > > diff --git a/automation/gitlab-ci/build.yaml > > b/automation/gitlab-ci/build.yaml > > index 35e224366f62..f12de00a164a 100644 > > --- a/automation/gitlab-ci/build.yaml > > +++ b/automation/gitlab-ci/build.yaml > > @@ -12,6 +12,12 @@ > > - '*/*.log' > > when: always > > needs: [] > > + rules: > > + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY > > + when: always > > + - if: $SELECTED_JOBS_ONLY > > + when: never > > + - when: on_success > > > > .gcc-tmpl: > > variables: &gcc > > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml > > index c21a37933881..93632f1f9204 100644 > > --- a/automation/gitlab-ci/test.yaml > > +++ b/automation/gitlab-ci/test.yaml > > @@ -1,6 +1,11 @@ > > .test-jobs-common: > > stage: test > > image: ${XEN_REGISTRY}/${CONTAINER} > > + rules: > > + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY > > + - if: $SELECTED_JOBS_ONLY > > + when: never > > + - when: on_success > > > > .arm64-test-needs: &arm64-test-needs > > - alpine-3.18-arm64-rootfs-export > > @@ -99,6 +104,9 @@ > > - '*.dtb' > > when: always > > rules: > > + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY > > + - if: $SELECTED_JOBS_ONLY > > + when: never > > - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" > > tags: > > - xilinx > > @@ -117,6 +125,9 @@ > > - '*.log' > > when: always > > rules: > > + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY > > + - if: $SELECTED_JOBS_ONLY > > + when: never > > - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" > > tags: > > - xilinx > > @@ -137,6 +148,9 @@ > > - '*.log' > > when: always > > rules: > > + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY > > + - if: $SELECTED_JOBS_ONLY > > + when: never > > - if: $QUBES_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" > > tags: > > - qubes-hw2 > > -- > > git-series 0.9.1 > > -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab Attachment:
signature.asc
|
![]() |
Lists.xenproject.org is hosted with RackSpace, monitoring our |