Feature request: Require (only) one specific job in matrix to finish for other dependent job

[EwoutH]

Select Topic Area

Product Feedback

Body

In GitHub Actions, imagine I have a build configuration like this, which builds my program on 3 different OSes and tests it on Ubuntu:

name: build-and-test

jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]


  test:
    runs-on: ubuntu-latest
    needs: build

I would like the test job to depent on the Ubuntu run from the build job. However, now it depends on all three. How can I specify that the test job only needs the ubuntu run from the build job to finish before it starts running, and not all three?

TL;DR: I would like the test job to run as soon as the Ubuntu build job is finished, and not wait on the macOS and Windows ones.

[vaind]

I've been missing this for a long time now. My use-case is a bit more complicated though - I have multiple matrix jobs and I'd like the dependent jobs started as soon as their dependency from another matrix finishes. TLDR: it would be great if we could use the matrix syntax with its brevity and flexibility and yet still be able to start subsequent jobs as soon as their dependency is done building, thus decreasing the overall time to feedback for the developer as well as the total CI runtime (depends on use case)

To update your example @EwoutH, I'd like to see something like this working:

name: build-and-test

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        variant: [develop, beta, master]
    steps:
      - uses: actions/checkout@v3
      - run: ./build ${{ matrix.variant }}
      - uses: actions/upload-artifact@v3
        with:
          name: test-app
          path: test-app

  test:
    runs-on: ubuntu-latest
    needs: [build:${{ matrix.variant }}]	# The magic needs to happen here
    strategy:
      matrix:
        variant: [develop, beta, master]
    steps:
      - uses: actions/checkout@v3[
      - uses: actions/download-artifact@v3
        with:
          name: test-app
      - run: ./test-app 

The gist is each matrix job should expose a name based on all the variables in the matrix. In this case, there's just one but it should be generalized for any number of variables. There are other possibilities for the exact syntax of the job name (and thus what should be defined in the needs section on dependants), e.g. build:variant=develop. build:variant=beta. This could be generalized further and is not limited to a single level of dependency.

Another example of the possible syntax is based on how GitLab does it: https://docs.gitlab.com/ee/ci/jobs/job_control.html#fetch-artifacts-from-a-parallelmatrix-job

ruby:
  image: ruby:${RUBY_VERSION}
  parallel:
    matrix:
      - RUBY_VERSION: ["2.5", "2.6", "2.7", "3.0", "3.1"]
        PROVIDER: [aws, gcp]
  script: bundle install

deploy:
  image: ruby:2.7
  stage: deploy
  dependencies:
    - "ruby: [2.7, aws]"
  script: echo hello
  environment: production

[sergeykranga]

We also have a similar use case where our workflow times would be sped up by 2-3 times if this was implemented. To me this is an obvious feature to add when you are dealing with matrices in more advanced workflows.

So it would be nice to get at least an answer if this is even considered to be implemented?

[bkona-alopa]

This feature is a must - this increases the performance
It would be great, if this is added..!

[cfoch]

I can do this in Jenkins.

[h2oearth]

I agree that this feature is a must-have!

[Mikifarber1]

I also need this feature!

[paul-gilber]

I also need this feature

[SanderVocke]

I need this feature too.

[xhoto]

I also need this feature!!!

[snps-midhunn]

+1

[snps-swamyt]

I am also in need of this feature.

[lpeabody]

I'm Spartacus.

...and like everyone else here, this feature would be killer.

[infogulch]

Lord have mercy please just upvote, thumbs up, and subscribe to the discussion instead of spamming 50 people just to write "pLuS oNe MeE tOo I wOuLd LiKe ThIs FeAtUrE" for the 30th time.

[ChristopherHX]

FYI The workflow could be rewritten into one workflow and one reusable workflow without relying on missing features.
Just in case you end up here and don't want to wait another year for an official answer.

These are just sketches of the OP rewritten using reusable workflows.

.github/workflows/build-and-test-template.yml

name: build-and-test-template
on:
  workflow_call:
    inputs:
      os:
        type: string
        required: true
jobs:
  build:
# add the os from the outer matrix via inputs
    runs-on: ${{ inputs.os }}

  test:
# If you remove this, then every matrix leg has a test job
    if: inputs.os == 'ubuntu-latest'
# ${{ inputs.os }} can be used here too
    runs-on: ubuntu-latest
    needs: build

.github/workflows/build-and-test.yml

name: build-and-test
on: push
jobs:
  build:
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
    uses: ./.github/workflows/build-and-test-template.yml
    with:
      os: ${{ matrix.os }}
    # If you might want to use a secret in the build or test job then you would need to pass secrets, inherit disables secret isolation
    secrets: inherit

You might need to configure permission properties depending on what you do with github.token

This might end up a bit degenerated in the workflow UI, but should work like that using existing features.

[ChristopherHX]

Yes I have a needs: build in my example, but job build is no longer a matrix job so doesn't wait for other Operating System builds.

The matrix is now defined over both build and test jobs via the uses key, so they run sequencially for each matrix configuation.

Eventually it was not a good idea to call two jobs build in two workflows...

Actually I was still to lazy to refactor my own workflows to use this method, since at the beginning of reusable workflowa you couldn't apply the matrix option...

[ChristopherHX]

This still works for me, job linux test runs before macos and windows finished... https://github.com/ChristopherHX/newcomposite-sample/actions/runs/8853596495/job/24314815591

[neilmayhew]

Ah! Now I understand. Thanks for explaining, and for the screen shot. Looks like a nice solution, and I think I'll be able to use the same technique on my project.

[roy-work]

I'm going to give this a try, but I'd love to know what the Summary tab for your example looks like.

[ChristopherHX]

You can technically just follow my link to my public test repo to see it there

The overview doesn't look good, you can expect seeing skipped jobs when you only have a test job for linux

[Paebbels]

Hint: This feature is supported by GitLab CI.

---

Suggestions:
To avoid complex syntax for job names as a string, the syntax of yaml could be used.
A simple string is equivalent to a job id. If needed job_name could be provided too.
If matrix is set in addition, job_id must be present. Each mapping describes a matrix variable/value combination.

needs:
  - job_id: build
    matrix:
      os: ubuntu

As jobs can be nested due to reusable workflows, that hierarchy should be expressible. In the following example, the job group ubuntu refers to the job build within a reusable workflow, which is a matrix job for multiple gcc versions. In this case, an outer job can continue when the build is complete, but it doesn't need to wait for a test job (ubuntu/test).

needs:
  - job_id: ubuntu/build
    matrix:
      gcc: 14

---

Some possible workaround (but creating other problems).

1. Switch from 2 jobs with identical matrix configurations to a single matrix calling a reusable workflow with 2 jobs. Matrix parameters are handed over via workflow inputs.
   Pros:

   • Job A2 is started after A1 finishes, without waiting for variants B1,C1,D1 to finish.

   Cons:

   • This enhances speed for A1->A2, B1->B2, etc. but a common job just needing a single matrix combination to finish still wait on A2, B2, C2, D2.
   • Dependencies are not displayed correctly in the graph. When A2 starts, a new job pops up.
   • More groups are displayed in the job navigation bar. Instead of 1:A,B,C,D and 2:A,B,C,D we have now A:1,2, B:1,2, C:1,2, D:1,2.

2. The job (job group) depending on a single matrix combination could be a reusable workflow, which is called only for a certain condition using an if.
   Cons:

   • Increased complexity
   • nesting of workflows is limited to 4 levels.
   • I suspect workflow outputs might get complicated or impossible with nesting.

[torsknod2]

I have a similar need.
I have a workflow, which has a matrix job to generate documentation in different formats.
For one of the formats, I need a consecutive matrix job, to generate further formats.

[mikepurvis]

Recently began learning GHA after years of GitLab and Jenkins— was quite surprised to find this feature missing.

[f-michaut]

I would also need this, I dont really care what the syntax will look like, even writting needs: "build (ubuntu-latest, Release, gcc)" (which is the name of the job in the UI) would be acceptable.