Setup prometheus + grafana

[severo]

Related to #2

• expose a /metrics endpoint using the Prometheus spec in the API, eg using https://github.com/prometheus/client_python - see Add metrics #258. Beware: cache and queue metrics removed after The API is unavailable in production #279
• Use a ServiceMonitor in the Helm chart: https://github.com/huggingface/tensorboard-launcher/blob/main/kube/templates/servicemonitor.yaml: see Add service monitor #260
• create a dashboard in grafana. The recommended process is to:
  • ensure Grafana can see the metrics: OK, at https://grafana.huggingface.tech/explore?orgId=1&left=%7B%22datasource%22:%22Prometheus%20EKS%20Hub%20Prod%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22instant%22:true,%22range%22:false,%22exemplar%22:false,%22expr%22:%22%7Bjob%3D%5C%22datasets-server-prod-api%5C%22,%20__name__%3D~%5C%22.%2B%5C%22%7D%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D
  • read the doc written by @McPatate
  • create a new JSON in https://github.com/huggingface/infra/blob/main/projects/monitoring/grafana/dashboards/hub/, copying one of the existing ones
  • create a PR: an ephemeral grafana will be available to test
  • tweak the dashboard in the grafana frontend: https://grafana.huggingface.tech/?orgId=1 -> save: will give an error but allow to download the JSON to paste into the PR
• add metrics about the cache and the queue. See discussion in The API is unavailable in production #279 and work on Create custom grafana dashboards #310

[severo]

OK, done here: https://grafana.huggingface.tech/d/SaHl2KX7z/datasets-server?orgId=1. Will be improved progressively

[severo]

I tried to re-enable the metrics with #298, since we know have indexes but had to revert immediately because the pod became unreachable. I'll try to:

• only introduce the starlette metrics (from https://github.com/perdy/starlette-prometheus/), without the cache+queue metrics from mongo - see fix: 🐛 disable cache and queue metrics for now #300
• try to add the queue metrics, but finding a way to test and quantify the additional queries to mongo, and optimize them before trying in production
• do the same with cache

[McPatate]

Would you want to go through your current monitoring stack together ? Maybe doing a bit of peer programming on this can help you out

[severo]

Beware: as we now (#304) have 3 replicas of the api app, I'm not sure how we should manage the starlette metrics: https://datasets-server.huggingface.tech/metrics will give the metrics for the particular node (and possibly also the particular uvicorn worker) that has been reached -> we have currently 9 x 3 = 27 parallel starlette apps (isn't it too much, by the way?)

[McPatate]

You may indeed not need that many starlette apps. Do you know what your total load looks like in req/s ?