Skip to content

[Mixins] Option for a custom cluster label #1651

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Apr 13, 2022
Merged

[Mixins] Option for a custom cluster label #1651

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
10b2095
Add custom clusterlabel to dashboards and recording rules
Whyeasy Apr 6, 2022
d6d4066
Use correct group for grouping metrics
Whyeasy Apr 6, 2022
058c906
Revert groups by labels
Whyeasy Apr 7, 2022
af70959
Add cluster label in alerts for recording rules
Whyeasy Apr 7, 2022
e9f6b09
Update CHANGELOG.md
Whyeasy Apr 7, 2022
ca1b6d6
Fix indentation config.libsonnet
Whyeasy Apr 7, 2022
7b32d0a
Use alert_aggreation at the obvious locations
Whyeasy Apr 8, 2022
a420133
rename clusterLabel and add namespaceMatcher
Whyeasy Apr 13, 2022
b1637d2
Merge branch 'main' into custom-cluster-label
Whyeasy Apr 13, 2022
d516f1b
Update CHANGELOG.md
pracucci Apr 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@

* [CHANGE] Dashboards: Remove per-user series legends from Tenants dashboard. #1605
* [CHANGE] Dashboards: Show in-memory series and the per-user series limit on Tenants dashboard. #1613
* [ENHANCEMENT] Alerting rules: Add possibility to use a custom cluster label.
* [ENHANCEMENT] Dashboards: Add possibility to use a custom cluster label.
* [ENHANCEMENT] Recording rules: Add possibility to use a custom cluster label.
Copy link
Collaborator

@pracucci pracucci Apr 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* [ENHANCEMENT] Alerting rules: Add possibility to use a custom cluster label.
* [ENHANCEMENT] Dashboards: Add possibility to use a custom cluster label.
* [ENHANCEMENT] Recording rules: Add possibility to use a custom cluster label.
* [ENHANCEMENT] Added `per_cluster_label` support to allow to change the label name used to differentiate between Kubernetes clusters. #1651

Copy link
Contributor Author

@Whyeasy Whyeasy Apr 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

* [BUGFIX] Dashboards: Fix "Failed evaluation rate" panel on Tenants dashboard. #1629

### Jsonnet
Expand Down
2 changes: 1 addition & 1 deletion operations/mimir-mixin/alerts/alerts.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -435,7 +435,7 @@
alert: $.alertName('ProvisioningTooManyWrites'),
// 80k writes / s per ingester max.
expr: |||
avg by (%(alert_aggregation_labels)s) (cluster_namespace_%(per_instance_label)s:cortex_ingester_ingested_samples_total:rate1m) > 80e3
avg by (%(alert_aggregation_labels)s) (%(clusterLabel)s_namespace_%(per_instance_label)s:cortex_ingester_ingested_samples_total:rate1m) > 80e3
||| % $._config,
'for': '15m',
labels: {
Expand Down
6 changes: 3 additions & 3 deletions operations/mimir-mixin/alerts/blocks.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,13 @@
(max by(%(alert_aggregation_labels)s, %(per_instance_label)s) (thanos_objstore_bucket_last_successful_upload_time{job=~".+/ingester.*"}) > 0)
and
# Only if the ingester has ingested samples over the last 4h.
(max by(%(alert_aggregation_labels)s, %(per_instance_label)s) (max_over_time(cluster_namespace_%(per_instance_label)s:cortex_ingester_ingested_samples_total:rate1m[4h])) > 0)
(max by(%(alert_aggregation_labels)s, %(per_instance_label)s) (max_over_time(%(clusterLabel)s_namespace_%(per_instance_label)s:cortex_ingester_ingested_samples_total:rate1m[4h])) > 0)
and
# Only if the ingester was ingesting samples 4h ago. This protects from the case the ingester instance
# had ingested samples in the past, then no traffic was received for a long period and then it starts
# receiving samples again. Without this check, the alert would fire as soon as it gets back receiving
# samples, while the a block shipping is expected within the next 4h.
(max by(%(alert_aggregation_labels)s, %(per_instance_label)s) (max_over_time(cluster_namespace_%(per_instance_label)s:cortex_ingester_ingested_samples_total:rate1m[1h] offset 4h)) > 0)
(max by(%(alert_aggregation_labels)s, %(per_instance_label)s) (max_over_time(%(clusterLabel)s_namespace_%(per_instance_label)s:cortex_ingester_ingested_samples_total:rate1m[1h] offset 4h)) > 0)
||| % $._config,
labels: {
severity: 'critical',
Expand All @@ -37,7 +37,7 @@
expr: |||
(max by(%(alert_aggregation_labels)s, %(per_instance_label)s) (thanos_objstore_bucket_last_successful_upload_time{job=~".+/ingester.*"}) == 0)
and
(max by(%(alert_aggregation_labels)s, %(per_instance_label)s) (max_over_time(cluster_namespace_%(per_instance_label)s:cortex_ingester_ingested_samples_total:rate1m[4h])) > 0)
(max by(%(alert_aggregation_labels)s, %(per_instance_label)s) (max_over_time(%(clusterLabel)s_namespace_%(per_instance_label)s:cortex_ingester_ingested_samples_total:rate1m[4h])) > 0)
||| % $._config,
labels: {
severity: 'critical',
Expand Down
7 changes: 5 additions & 2 deletions operations/mimir-mixin/config.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,12 @@
overrides_exporter: 'overrides-exporter',
},

// Custom Cluster label to use in dashboards.
clusterLabel: 'cluster',
Copy link
Collaborator

@pracucci pracucci Apr 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should introduce this, but we should just keep using job_labels and cluster_labels everywhere. For example, cluster_labels is already used to build alert_aggregation_labels which is then used in alerts and recording rules. However, not all of them are using it so I think we should just keep working to make sure they're used everywhere (see few other comments I left).

Copy link
Contributor Author

@Whyeasy Whyeasy Apr 8, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will start working on using alert_aggregation_labels.

But what's your idea on the approach we should take here:

mimir/operations/mimir-mixin/dashboards/dashboard-utils.libsonnet

Line 57 in ca1b6d6

.addMultiTemplate('cluster', 'cortex_build_info', '%s' % $._config.clusterLabel)
. Because this selector should be updated accordingly.

Copy link
Collaborator

@pracucci pracucci Apr 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the difficulty of getting rid of this config field at all. Ok, let's keep it. Can I just ask you to rename clusterLabel to per_cluster_label to keep it consistent with per_instance_label and per_node_label, please?

The comment could be something like:

// The label used to differentiate between different Kubernetes clusters.


// Grouping labels, to uniquely identify and group by {jobs, clusters}
job_labels: ['cluster', 'namespace', 'job'],
cluster_labels: ['cluster', 'namespace'],
job_labels: [$._config.clusterLabel, 'namespace', 'job'],
cluster_labels: [$._config.clusterLabel, 'namespace'],

cortex_p99_latency_threshold_seconds: 2.5,

Expand Down
48 changes: 24 additions & 24 deletions operations/mimir-mixin/dashboards/alertmanager.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,11 @@ local utils = import 'mixin-utils/utils.libsonnet';
})
.addPanel(
$.panel('Total alerts') +
$.statPanel('sum(cluster_job_%s:cortex_alertmanager_alerts:sum{%s})' % [$._config.per_instance_label, $.jobMatcher($._config.job_names.alertmanager)], format='short')
$.statPanel('sum(%s_job_%s:cortex_alertmanager_alerts:sum{%s})' % [$._config.clusterLabel, $._config.per_instance_label, $.jobMatcher($._config.job_names.alertmanager)], format='short')
)
.addPanel(
$.panel('Total silences') +
$.statPanel('sum(cluster_job_%s:cortex_alertmanager_silences:sum{%s})' % [$._config.per_instance_label, $.jobMatcher($._config.job_names.alertmanager)], format='short')
$.statPanel('sum(%s_job_%s:cortex_alertmanager_silences:sum{%s})' % [$._config.clusterLabel, $._config.per_instance_label, $.jobMatcher($._config.job_names.alertmanager)], format='short')
)
.addPanel(
$.panel('Tenants') +
Expand All @@ -29,11 +29,11 @@ local utils = import 'mixin-utils/utils.libsonnet';
$.queryPanel(
[
|||
sum(cluster_job:cortex_alertmanager_alerts_received_total:rate5m{%s})
sum(%s_job:cortex_alertmanager_alerts_received_total:rate5m{%s})
-
sum(cluster_job:cortex_alertmanager_alerts_invalid_total:rate5m{%s})
||| % [$.jobMatcher($._config.job_names.alertmanager), $.jobMatcher($._config.job_names.alertmanager)],
'sum(cluster_job:cortex_alertmanager_alerts_invalid_total:rate5m{%s})' % $.jobMatcher($._config.job_names.alertmanager),
sum(%s_job:cortex_alertmanager_alerts_invalid_total:rate5m{%s})
||| % [$._config.clusterLabel, $.jobMatcher($._config.job_names.alertmanager), $._config.clusterLabel, $.jobMatcher($._config.job_names.alertmanager)],
'sum(%s_job:cortex_alertmanager_alerts_invalid_total:rate5m{%s})' % [$._config.clusterLabel, $.jobMatcher($._config.job_names.alertmanager)],
],
['success', 'failed']
)
Expand All @@ -46,11 +46,11 @@ local utils = import 'mixin-utils/utils.libsonnet';
$.queryPanel(
[
|||
sum(cluster_job_integration:cortex_alertmanager_notifications_total:rate5m{%s})
sum(%s_job_integration:cortex_alertmanager_notifications_total:rate5m{%s})
-
sum(cluster_job_integration:cortex_alertmanager_notifications_failed_total:rate5m{%s})
||| % [$.jobMatcher($._config.job_names.alertmanager), $.jobMatcher($._config.job_names.alertmanager)],
'sum(cluster_job_integration:cortex_alertmanager_notifications_failed_total:rate5m{%s})' % $.jobMatcher($._config.job_names.alertmanager),
sum(%s_job_integration:cortex_alertmanager_notifications_failed_total:rate5m{%s})
||| % [$._config.clusterLabel, $.jobMatcher($._config.job_names.alertmanager), $._config.clusterLabel, $.jobMatcher($._config.job_names.alertmanager)],
'sum(%s_job_integration:cortex_alertmanager_notifications_failed_total:rate5m{%s})' % [$._config.clusterLabel, $.jobMatcher($._config.job_names.alertmanager)],
],
['success', 'failed']
)
Expand All @@ -61,13 +61,13 @@ local utils = import 'mixin-utils/utils.libsonnet';
[
|||
(
sum(cluster_job_integration:cortex_alertmanager_notifications_total:rate5m{%s}) by(integration)
sum(%s_job_integration:cortex_alertmanager_notifications_total:rate5m{%s}) by(integration)
-
sum(cluster_job_integration:cortex_alertmanager_notifications_failed_total:rate5m{%s}) by(integration)
sum(%s_job_integration:cortex_alertmanager_notifications_failed_total:rate5m{%s}) by(integration)
) > 0
or on () vector(0)
||| % [$.jobMatcher($._config.job_names.alertmanager), $.jobMatcher($._config.job_names.alertmanager)],
'sum(cluster_job_integration:cortex_alertmanager_notifications_failed_total:rate5m{%s}) by(integration)' % $.jobMatcher($._config.job_names.alertmanager),
||| % [$._config.clusterLabel, $.jobMatcher($._config.job_names.alertmanager), $._config.clusterLabel, $.jobMatcher($._config.job_names.alertmanager)],
'sum(%s_job_integration:cortex_alertmanager_notifications_failed_total:rate5m{%s}) by(integration)' % [$._config.clusterLabel, $.jobMatcher($._config.job_names.alertmanager)],
],
['success - {{ integration }}', 'failed - {{ integration }}']
)
Expand Down Expand Up @@ -104,15 +104,15 @@ local utils = import 'mixin-utils/utils.libsonnet';
.addPanel(
$.panel('Per %s alerts' % $._config.per_instance_label) +
$.queryPanel(
'sum by(%s) (cluster_job_%s:cortex_alertmanager_alerts:sum{%s})' % [$._config.per_instance_label, $._config.per_instance_label, $.jobMatcher($._config.job_names.alertmanager)],
'sum by(%s) (%s_job_%s:cortex_alertmanager_alerts:sum{%s})' % [$._config.clusterLabel, $._config.per_instance_label, $._config.per_instance_label, $.jobMatcher($._config.job_names.alertmanager)],
'{{%s}}' % $._config.per_instance_label
) +
$.stack
)
.addPanel(
$.panel('Per %s silences' % $._config.per_instance_label) +
$.queryPanel(
'sum by(%s) (cluster_job_%s:cortex_alertmanager_silences:sum{%s})' % [$._config.per_instance_label, $._config.per_instance_label, $.jobMatcher($._config.job_names.alertmanager)],
'sum by(%s) (%s_job_%s:cortex_alertmanager_silences:sum{%s})' % [$._config.clusterLabel, $._config.per_instance_label, $._config.per_instance_label, $.jobMatcher($._config.job_names.alertmanager)],
'{{%s}}' % $._config.per_instance_label
) +
$.stack
Expand Down Expand Up @@ -205,11 +205,11 @@ local utils = import 'mixin-utils/utils.libsonnet';
$.queryPanel(
[
|||
sum(cluster_job:cortex_alertmanager_state_replication_total:rate5m{%s})
sum(%s_job:cortex_alertmanager_state_replication_total:rate5m{%s})
-
sum(cluster_job:cortex_alertmanager_state_replication_failed_total:rate5m{%s})
||| % [$.jobMatcher($._config.job_names.alertmanager), $.jobMatcher($._config.job_names.alertmanager)],
'sum(cluster_job:cortex_alertmanager_state_replication_failed_total:rate5m{%s})' % $.jobMatcher($._config.job_names.alertmanager),
sum(%s_job:cortex_alertmanager_state_replication_failed_total:rate5m{%s})
||| % [$._config.clusterLabel, $.jobMatcher($._config.job_names.alertmanager), $._config.clusterLabel, $.jobMatcher($._config.job_names.alertmanager)],
'sum(%s_job:cortex_alertmanager_state_replication_failed_total:rate5m{%s})' % [$._config.clusterLabel, $.jobMatcher($._config.job_names.alertmanager)],
],
['success', 'failed']
)
Expand All @@ -219,11 +219,11 @@ local utils = import 'mixin-utils/utils.libsonnet';
$.queryPanel(
[
|||
sum(cluster_job:cortex_alertmanager_partial_state_merges_total:rate5m{%s})
sum(%s_job:cortex_alertmanager_partial_state_merges_total:rate5m{%s})
-
sum(cluster_job:cortex_alertmanager_partial_state_merges_failed_total:rate5m{%s})
||| % [$.jobMatcher($._config.job_names.alertmanager), $.jobMatcher($._config.job_names.alertmanager)],
'sum(cluster_job:cortex_alertmanager_partial_state_merges_failed_total:rate5m{%s})' % $.jobMatcher($._config.job_names.alertmanager),
sum(%s_job:cortex_alertmanager_partial_state_merges_failed_total:rate5m{%s})
||| % [$._config.clusterLabel, $.jobMatcher($._config.job_names.alertmanager), $._config.clusterLabel, $.jobMatcher($._config.job_names.alertmanager)],
'sum(%s_job:cortex_alertmanager_partial_state_merges_failed_total:rate5m{%s})' % [$._config.clusterLabel, $.jobMatcher($._config.job_names.alertmanager)],
],
['success', 'failed']
)
Expand Down
18 changes: 9 additions & 9 deletions operations/mimir-mixin/dashboards/dashboard-utils.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,17 +54,17 @@ local utils = import 'mixin-utils/utils.libsonnet';
if $._config.singleBinary
then d.addMultiTemplate('job', 'cortex_build_info', 'job')
else d
.addMultiTemplate('cluster', 'cortex_build_info', 'cluster')
.addMultiTemplate('namespace', 'cortex_build_info{cluster=~"$cluster"}', 'namespace')
.addMultiTemplate('cluster', 'cortex_build_info', '%s' % $._config.clusterLabel)
.addMultiTemplate('namespace', 'cortex_build_info{%s=~"$cluster"}' % $._config.clusterLabel , 'namespace')
else
if $._config.singleBinary
then d.addTemplate('job', 'cortex_build_info', 'job')
else d
.addTemplate('cluster', 'cortex_build_info', 'cluster')
.addTemplate('namespace', 'cortex_build_info{cluster=~"$cluster"}', 'namespace'),
.addTemplate('cluster', 'cortex_build_info', '%s' % $._config.clusterLabel)
.addTemplate('namespace', 'cortex_build_info{%s=~"$cluster"}' % $._config.clusterLabel, 'namespace'),

addActiveUserSelectorTemplates()::
self.addTemplate('user', 'cortex_ingester_active_series{cluster=~"$cluster", namespace=~"$namespace"}', 'user'),
self.addTemplate('user', 'cortex_ingester_active_series{%s=~"$cluster", namespace=~"$namespace"}' % $._config.clusterLabel, 'user'),

addCustomTemplate(name, values, defaultIndex=0):: self {
templating+: {
Expand Down Expand Up @@ -99,17 +99,17 @@ local utils = import 'mixin-utils/utils.libsonnet';
jobMatcher(job)::
if $._config.singleBinary
then 'job=~"$job"'
else 'cluster=~"$cluster", job=~"($namespace)/(%s)"' % job,
else '%s=~"$cluster", job=~"($namespace)/(%s)"' % [$._config.clusterLabel, job],

namespaceMatcher()::
if $._config.singleBinary
then 'job=~"$job"'
else 'cluster=~"$cluster", namespace=~"$namespace"',
else '%s=~"$cluster", namespace=~"$namespace"' % $._config.clusterLabel,

jobSelector(job)::
if $._config.singleBinary
then [utils.selector.noop('cluster'), utils.selector.re('job', '$job')]
else [utils.selector.re('cluster', '$cluster'), utils.selector.re('job', '($namespace)/(%s)' % job)],
then [utils.selector.noop('%s' % $._config.clusterLabel), utils.selector.re('job', '$job')]
else [utils.selector.re('%s' % $._config.clusterLabel, '$cluster'), utils.selector.re('job', '($namespace)/(%s)' % job)],

queryPanel(queries, legends, legendLink=null)::
super.queryPanel(queries, legends, legendLink) + {
Expand Down
4 changes: 2 additions & 2 deletions operations/mimir-mixin/dashboards/overrides.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ local utils = import 'mixin-utils/utils.libsonnet';
datasource: '${datasource}',
targets: [
{
expr: 'max by(limit_name) (cortex_limits_defaults{cluster=~"$cluster",namespace=~"$namespace"})',
expr: 'max by(limit_name) (cortex_limits_defaults{%s=~"$cluster",namespace=~"$namespace"})' % $._config.clusterLabel,
instant: true,
legendFormat: '',
refId: 'A',
Expand Down Expand Up @@ -69,7 +69,7 @@ local utils = import 'mixin-utils/utils.libsonnet';
datasource: '${datasource}',
targets: [
{
expr: 'max by(user, limit_name) (cortex_limits_overrides{cluster=~"$cluster",namespace=~"$namespace",user=~"${tenant_id}"})',
expr: 'max by(user, limit_name) (cortex_limits_overrides{%s=~"$cluster",namespace=~"$namespace",user=~"${tenant_id}"})' % $._config.clusterLabel,
instant: true,
legendFormat: '',
refId: 'A',
Expand Down
13 changes: 7 additions & 6 deletions operations/mimir-mixin/dashboards/rollout-progress.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ local utils = import 'mixin-utils/utils.libsonnet';
(import 'dashboard-utils.libsonnet') {
local config = {
namespace_matcher: $.namespaceMatcher(),
clusterLabel: $._config.clusterLabel,
gateway_job_matcher: $.jobMatcher($._config.job_names.gateway),
gateway_write_routes_regex: 'api_(v1|prom)_push',
gateway_read_routes_regex: '(prometheus|api_prom)_api_v1_.+',
Expand Down Expand Up @@ -127,7 +128,7 @@ local utils = import 'mixin-utils/utils.libsonnet';

$.panel('Writes 99th latency') +
$.newStatPanel(|||
histogram_quantile(0.99, sum by (le) (cluster_job_route:cortex_request_duration_seconds_bucket:sum_rate{%(gateway_job_matcher)s, route=~"%(gateway_write_routes_regex)s"}))
histogram_quantile(0.99, sum by (le) (%(clusterLabel)s_job_route:cortex_request_duration_seconds_bucket:sum_rate{%(gateway_job_matcher)s, route=~"%(gateway_write_routes_regex)s"}))
||| % config, unit='s', thresholds=[
{ color: 'green', value: null },
{ color: 'orange', value: 0.2 },
Expand Down Expand Up @@ -178,7 +179,7 @@ local utils = import 'mixin-utils/utils.libsonnet';

$.panel('Reads 99th latency') +
$.newStatPanel(|||
histogram_quantile(0.99, sum by (le) (cluster_job_route:cortex_request_duration_seconds_bucket:sum_rate{%(gateway_job_matcher)s, route=~"%(gateway_read_routes_regex)s"}))
histogram_quantile(0.99, sum by (le) (%(clusterLabel)s_job_route:cortex_request_duration_seconds_bucket:sum_rate{%(gateway_job_matcher)s, route=~"%(gateway_read_routes_regex)s"}))
||| % config, unit='s', thresholds=[
{ color: 'green', value: null },
{ color: 'orange', value: 1 },
Expand Down Expand Up @@ -283,15 +284,15 @@ local utils = import 'mixin-utils/utils.libsonnet';
$.panel('Latency vs 24h ago') +
$.queryPanel([|||
1 - (
avg_over_time(histogram_quantile(0.99, sum by (le) (cluster_job_route:cortex_request_duration_seconds_bucket:sum_rate{%(gateway_job_matcher)s, route=~"%(gateway_write_routes_regex)s"} offset 24h))[1h:])
avg_over_time(histogram_quantile(0.99, sum by (le) (%(clusterLabel)s_job_route:cortex_request_duration_seconds_bucket:sum_rate{%(gateway_job_matcher)s, route=~"%(gateway_write_routes_regex)s"} offset 24h))[1h:])
/
avg_over_time(histogram_quantile(0.99, sum by (le) (cluster_job_route:cortex_request_duration_seconds_bucket:sum_rate{%(gateway_job_matcher)s, route=~"%(gateway_write_routes_regex)s"}))[1h:])
avg_over_time(histogram_quantile(0.99, sum by (le) (%(clusterLabel)s_job_route:cortex_request_duration_seconds_bucket:sum_rate{%(gateway_job_matcher)s, route=~"%(gateway_write_routes_regex)s"}))[1h:])
)
||| % config, |||
1 - (
avg_over_time(histogram_quantile(0.99, sum by (le) (cluster_job_route:cortex_request_duration_seconds_bucket:sum_rate{%(gateway_job_matcher)s, route=~"%(gateway_read_routes_regex)s"} offset 24h))[1h:])
avg_over_time(histogram_quantile(0.99, sum by (le) (%(clusterLabel)s_job_route:cortex_request_duration_seconds_bucket:sum_rate{%(gateway_job_matcher)s, route=~"%(gateway_read_routes_regex)s"} offset 24h))[1h:])
/
avg_over_time(histogram_quantile(0.99, sum by (le) (cluster_job_route:cortex_request_duration_seconds_bucket:sum_rate{%(gateway_job_matcher)s, route=~"%(gateway_read_routes_regex)s"}))[1h:])
avg_over_time(histogram_quantile(0.99, sum by (le) (%(clusterLabel)s_job_route:cortex_request_duration_seconds_bucket:sum_rate{%(gateway_job_matcher)s, route=~"%(gateway_read_routes_regex)s"}))[1h:])
)
||| % config], ['writes', 'reads']) + {
yaxes: $.yaxes({
Expand Down
2 changes: 1 addition & 1 deletion operations/mimir-mixin/dashboards/ruler.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ local utils = import 'mixin-utils/utils.libsonnet';
.addPanel(
$.panel('Per route p99 latency') +
$.queryPanel(
'histogram_quantile(0.99, sum by (route, le) (cluster_job_route:cortex_request_duration_seconds_bucket:sum_rate{%s, route=~"%s"}))' % [$.jobMatcher($._config.job_names.gateway), ruler_config_api_routes_re],
'histogram_quantile(0.99, sum by (route, le) (%s_job_route:cortex_request_duration_seconds_bucket:sum_rate{%s, route=~"%s"}))' % [$._config.clusterLabel, $.jobMatcher($._config.job_names.gateway), ruler_config_api_routes_re],
'{{ route }}'
) +
{ yaxes: $.yaxes('s') }
Expand Down
Loading