Scale the worker pods depending on prometheus metrics?

[severo]

We could scale the number of worker pods depending on:

• the size of the job queue
• the available resources
  These data are available in prometheus, and we could use them to autoscale the pods.

[severo]

https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

See #235 (comment)

[McPatate]

Also : https://github.com/kubernetes-sigs/prometheus-adapter

[severo]

First step: the nodes autoscale up to 20 nodes, while the pods must still be scaled manually

• https://github.com/huggingface/infra/pull/239/files
• feat: 🎸 adjust the prod resources #383

During the refresh (see https://twitter.com/severo_dev/status/1537425799564759041), we scaled the split workers to 56 pods, it worked very well: nearly 6 jobs finished per second (340 jobs/min). We had to refresh more than 30.000 splits. After some trials and errors, it took about 3 hours to refresh all the datasets. See https://grafana.huggingface.tech/d/SaHl2KX7z/datasets-server-admin-cache-and-queue?orgId=1&from=1655380800000&to=1655398800000

[McPatate]

I'm not sure I understand, it seems the autoscaling you have activated is related to k8s nodes, and not your pods. Have you configured a HorizontalPodAutoscaler resource for your dataset server or are you indeed scaling pods manually ?

[severo]

"scaling pods manually" <- this

That's why I wrote "First step". At least, the nodes autoscale and we don't spend too much when we only have few pods

[McPatate]

Ay ! thanks for clarifying !

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[severo]

Or depending on the CPU load of the current pods (idea by @lhoestq)

[severo]

moved to the internal tracker