workflow_job "queued" event fires for every matrix job at run start, even when max-parallel is holding them back.

[andrewdibiasio6]

🏷️ Discussion Type

Bug

💬 Feature/Topic Area

Workflow Configuration

Discussion Details

Summary

With strategy.max-parallel: 1, GitHub creates all matrix jobs when the run starts and sends a workflow_job event with action: queued for each of them immediately, even though only one can run at a time. Autoscalers that launch a runner per queued event (we use the terraform-aws-github-runner module) therefore launch a runner for every 'cell' up front. We haven't tested it, but this may affect other autoscalers that react to queued events too, for example Actions Runner Controller in its webhook-driven mode. The runners for later cells sit idle while earlier cells run, and may be reaped by idle-timeout before their cell becomes eligible.

Nothing in the event payload tells the autoscaler that the job is blocked by max-parallel rather than waiting for runner capacity.

What we observed (8 cells, max-parallel: 1, each cell sleeps 6 minutes, ephemeral runners)

• All 8 jobs have the same created_at. 8 queued webhook events were received within about 250 ms of each other.
• 8 runner launches followed about 36 seconds later (30 s configured webhook delay), one per event, before any cell had finished.
• queued → started per cell grows by about 6 minutes per cell (0 to 43 min), but the real runner wait after the previous cell finished was 0 to 0.5 min. The "queue time" is almost entirely earlier cells running.
• Half of the pre-launched runners never ran a job: they idled past the 20-minute minimum running time and were terminated by scale-down.

Contrast: matrix job that calls a reusable workflow

When the matrix job uses: a reusable workflow instead of having runs-on, the cells are not all created up front. Same max-parallel: 1, same 8 cells, same runner pool:

• 6 minutes after the run started, only cell 1 existed: 1 job, 1 queued event, 1 runner launch. (The direct matrix had 8 jobs, 8 events and 8 launches at that point.)
• Cell 2 was created 9 seconds after cell 1 completed, and not before.

So the same max-parallel produces very different queued timing depending on whether the matrix job has runs-on or calls a reusable workflow. We saw the same pattern in a separate scheduled workflow of ours that uses the reusable shape.

Reproduction

Attached workflows matrix-direct.yml and matrix-reusable.yml (plus _cell.yml). Run each with workflow_dispatch on a self-hosted runner pool. To see the events, register a repository or App webhook for workflow_job pointed at any request inspector, or compare job created_at with GET /repos/{owner}/{repo}/actions/runs/{run_id}/jobs.

matrix-direct.yml
matrix-reusable.yml
_cell.yml

Questions

1. Is queued for a matrix job that max-parallel is still holding back intended?
2. Is there, or could there be, a distinct status or event (like waiting for environment protection rules) for "created but not yet eligible to run", so autoscalers can avoid launching runners early?
3. Why does the reusable-workflow matrix behave differently from the direct matrix?

(Posted as a question first; happy to move to the roadmap as a feature request.)

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐