Re: [PATCH v1 3/3] automation: allow selecting individual jobs via CI variables

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Fri, 14 Feb 2025, Stefano Stabellini wrote:
> On Fri, 14 Feb 2025, Marek Marczykowski-Górecki wrote:
> > On Thu, Feb 13, 2025 at 05:36:47PM -0800, Stefano Stabellini wrote:
> > > On Thu, 13 Feb 2025, Marek Marczykowski-Górecki wrote:
> > > > Debugging sometimes involves running specific jobs on different
> > > > versions. It's useful to easily avoid running all of the not interesting
> > > > ones (for given case) to save both time and CI resources. Doing so used
> > > > to require changing the yaml files, usually in several places.
> > > > Ease this step by adding SELECTED_JOBS_ONLY variable that takes a regex.
> > > > Note that one needs to satisfy job dependencies on their own (for
> > > > example if a test job needs a build job, that specific build job
> > > > needs to be included too).
> > > > 
> > > > The variable can be specified via Gitlab web UI when scheduling a
> > > > pipeline, but it can be also set when doing git push directly:
> > > > 
> > > >     git push -o ci.variable=SELECTED_JOBS_ONLY="/job1|job2/"
> > > > 
> > > > More details at 
> > > > https://docs.gitlab.co.jp/ee/user/project/push_options.html
> > > > 
> > > > The variable needs to include regex for selecting jobs, including
> > > > enclosing slashes.
> > > > A coma/space separated list of jobs to select would be friendlier UX,
> > > > but unfortunately that is not supported:
> > > > https://gitlab.com/gitlab-org/gitlab/-/issues/209904 (note the proposed
> > > > workaround doesn't work for job-level CI_JOB_NAME).
> > > > On the other hand, the regex is more flexible (one can select for
> > > > example all arm32 jobs).
> > > 
> > > I was trying to find workarounds so that we could also support the
> > > simple list of comma-separated jobs you mentioned, but I couldn't find
> > > an easy way to do that.
> > > 
> > > However, one thing we can do is to support writing SELECTED_JOBS_ONLY in
> > > .gitlab-ci.yml as a commit in xen.git, for people that don't know or
> > > don't remember how to set ci.variable using the git command line.
> > 
> > You can always do it, in `variables` setting AFAIR.
> > 
> > > Given that this is for testing, I think it would be no problem to adding
> > > a special commit on top of your tree. We are just trying to make it
> > > easier compared to having to manually delete the list of jobs we don't
> > > need.
> > 
> > Yes, manually delete was awful. In practice I usually added always-false
> > rules, but still.
> > 
> > > But even with the special commit, I couldn't think of an easy way to
> > > achieve the nicer comma-separated list of jobs...
> > > 
> > > 
> > > > Signed-off-by: Marek Marczykowski-Górecki 
> > > > <marmarek@xxxxxxxxxxxxxxxxxxxxxx>
> > > > ---
> > > > This probably wants documenting beyond this commit message. I don't
> > > > think we have any CI-related docs anywhere, do we? Some new file in
> > > > docs/misc?
> > > 
> > > Yes please :-)
> > > 
> > > It would be also worth documenting how to create a simple pipeline by
> > > removing everything that you don't need for a single test
> > 
> > You mean how to find what jobs you need?
> > 
> > > > And also, it's possible to extend web ui for starting pipelines to
> > > > include pre-defined variables. I use it in qubes here if you want to
> > > > see:
> > > > https://gitlab.com/QubesOS/qubes-continuous-integration/-/pipelines/new
> > > 
> > > I don't have access to this
> > 
> > Oh, sorry. Screenshot attached.
> > 
> > And its definition looks like this:
> > https://gitlab.com/QubesOS/qubes-continuous-integration/-/blob/main/.gitlab-ci.yml?ref_type=heads#L15-26
> 
> Actually this gave me an idea. What if we flip the default? Normally we
> want to run all the jobs.
> 
> When doing development, we typically want to run one specific job
> attached to the work we are currently doing. This patch introduces a
> blacklist, but it looks like we want a whitelist instead?
> 
> Wouldn't it be easier to say: "delete everything except for
> adl-smoke-x86-64-dom0pvh-hvm-gcc-debug"
> 
> Then we can arrange it so adl-smoke-x86-64-dom0pvh-hvm-gcc-debug and its
> dependencies are left enabled. Everything else is disabled?

Sorry I sent it too fast without thinking. This patch is already
implementing the whitelist. The only thing we are really missing is the
automatic dependency enablement, but I don't know how that could be
implemented.