Dynamic resource assignment to runners

[Brenner87]

Hello. I'm currently working on PoC for migrating from on prem Bitbucket and Jenkins to GitHub.
Playing with actions-runner-controller and faced with the problem. Some of our teams require very huge resource amount for their runners while another teams need very few resources.
Currently I've created runnerDeployment with following resource limits:

      resources:
        limits:
          cpu: "8.0"
          memory: "40Gi"
        requests:
          cpu: "1.0"
          memory: "6Gi"

This doesn't solve the problem, because if k8s node do not have enough resources, the runner would be destroyed.
Is there any way of dynamic resource management?

1. Is there an ability to request resource amount from build?
2. Is there an ability to request resources on runner creation?
3. May I have more then 1 runner deployments with 1 controller? Would they work correct?

I tried to create 2 different runnerDeployments with 1 controller. But it doesn't seem like autoscaler work correct.

I didn't find answers to this questions in documentation. Maybe I missed something and someone can point me to correct place.

[kevholmes]

The standard way to manage this @Brenner87 is to create multiple runner types with their own HRA and RunnerDeployment, and add them as Runner Groups in your GH instance. Users can then request these via the runs-on: label in their workflows. You can add these runner groups into an Org or repo individually to limit who can access the types. For instance maybe most teams do not need 8 cores / 40 Gi per Job so you do not make that Runner Group (at least one RunnerDeployment) available to them.

[Brenner87]

Thank you @kevholmes. Sounds reasonable. Going to follow this way:)

[alexklibisz]

@Brenner87 We solved this as @kevholmes discussed. I also miss the ability to specify custom per-job resources when using the Jenkins Kubernetes plugin. I think this is the only thing I miss about Jenkins.

Another downside of the multi-hra approach, in its most default implementation, is that you have to keep at least one runner w/ non-trivial resources running at all times. There is some info in the docs about configuring a runnerdeployment to scale to zero. I haven't taken the time to fully understand it yet.

[roulettedares]

what if we could dynamically set the runner pod resources via workflow labels? example workflow:

big-cpu-job:
  runs-on: [self-hosted, 23cpu-7g]

new setRequestsBasedOnLabels in the RunnerSet CR

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  name: myorg-use-labels-to-set-requests
spec:
  setRequestsBasedOnLabels:
    enabled: true
    maxPodCpus: 30
    maxPodMemory: 10Gi

or even a new DynamicRequestsRunnerSet custom resource to keep things tidy and secure

[Brenner87]

Thank you, will try.

[alexklibisz]

@Brenner87 i don't think this functionality exists. I think @joshuasimon-taulia was just proposing an idea for a possible API

I think it would be a neat feature

[roulettedares]

someone smarter than me would have to think about security and resource management constraints. ex: the total resource consumption allow by all pods in the dynamic label runnerset, pod service account bindings, etc

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬