metrics cardinality for ghalistener

[christophermichaeljohnston]

Remove runner_id and runner_name from metrics as these are unique for each ephemeral runner cause a new metric for every action.

fixes #3670

[christophermichaeljohnston]

job_workflow_ref also causes horrible high cardinality so I removed it

[christophermichaeljohnston]

The result of this are metrics that function as actual counters:

gha_completed_jobs_total{enterprise="",event_name="push",job_name="test",job_result="succeeded",organization="...",repository="..."} 3

[IanMoroney]

@nikola-jokic , this is another cardinality fix for metrics.
Could it be reviewed and merged?

[jameshounshell]

If possible this would be great if it could be implemented as a command line flag to change what labels are included in the metric. I think that runner_id and runner_name should be removed by default, but I would like to keep job_workflow_ref.

This issue demonstrates how we could have a sane default of labels (I'll concede job_workflow_ref to be off) but I'd like to be able to enable it for my use case.

Another great use case is reducing the cardinality of the histograms gha_job_execution_duration and gha_job_startup_duration which currently have 48 buckets. It might be good to be able to supply our own buckets until a consensus is reached about fewer, more reasonable, buckets.

edit: The old runners included job_workflow_ref and though high'ish cardinaility there are only a finite amount of workflow names, not increasing forever like runner_id. re: #3185

[jameshounshell]

Small update, last week I deployed code identical to this (minus removing job_workflow_ref), plus the bucket fixes and it's been working nicely.

[nikola-jokic]

Hey, I think this is a very good addition. I agree, these labels are just causing problems. Thanks for testing it @jameshounshell !

[rekha-prakash-maersk]

@nikola-jokic It would be really helpful in terms of user experience and performance if this could be added to next release ? we run the SelfHosted runner in large scale and the experience is not great because the metrics are missing often due to high memory usage of listener pod and listeners restarting few times day due to cpu and memory pileup, though I have allocated more resources to it.

[nikola-jokic]

Hey @rekha-prakash-maersk,

That is why I fixed the linting errors myself, so I can be sure this will go in for the next release ☺️

[nikola-jokic]

LGTM

[rekha-prakash-maersk]

@nikola-jokic awesome, thanks! can't wait for next release!!

[jameshounshell]

Sorry I was bogged down and missed this. I still wanted to keep job_workflow_ref despite it's high cardinality (not infinite). I'll work up a PR to make it an optional label. Maybe we can discuss it's merits on that PR.

[nikola-jokic]

Hey @jameshounshell,

We are starting to work on metrics customizations. We are already designing a solution so please wait for that and we would appreciate your feedback!