Prometheus ClassCastException while scraping (Micrometer 1.13 / SB 3.3)

[knoobie]

Describe the bug
Multiple Methods annotated with @Timed (or creating a Timer by hand) with the same value to group them together throws a ClassCastException once histogram is enabled on some - but not on all.

java.lang.ClassCastException: class io.prometheus.metrics.model.snapshots.SummarySnapshot$SummaryDataPointSnapshot cannot be cast to class io.prometheus.metrics.model.snapshots.HistogramSnapshot$HistogramDataPointSnapshot (io.prometheus.metrics.model.snapshots.SummarySnapshot$SummaryDataPointSnapshot and io.prometheus.metrics.model.snapshots.HistogramSnapshot$HistogramDataPointSnapshot are in unnamed module of loader 'app')

	at io.prometheus.metrics.expositionformats.OpenMetricsTextFormatWriter.writeClassicHistogramBuckets(OpenMetricsTextFormatWriter.java:125)
	at io.prometheus.metrics.expositionformats.OpenMetricsTextFormatWriter.writeHistogram(OpenMetricsTextFormatWriter.java:120)
	at io.prometheus.metrics.expositionformats.OpenMetricsTextFormatWriter.write(OpenMetricsTextFormatWriter.java:72)
	at io.micrometer.prometheusmetrics.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:167)
	at io.micrometer.prometheusmetrics.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:163)
	at io.micrometer.prometheusmetrics.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:136)

Environment

• Micrometer version: 1.13 (bundled in Spring Boot 3.3)
• Micrometer registry: Prometheus

To Reproduce

Smallest example by Andy Wilkinson (wilkinsona) (don't wanna spam tag him)

package com.example;

import io.micrometer.core.instrument.Timer;
import io.micrometer.core.instrument.Timer.Sample;
import io.micrometer.prometheusmetrics.PrometheusConfig;
import io.micrometer.prometheusmetrics.PrometheusMeterRegistry;

public class Reproducer {

	public static void main(String[] args) throws InterruptedException {
		PrometheusMeterRegistry registry = new PrometheusMeterRegistry(new PrometheusConfig() {
			
			@Override
			public String get(String key) {
				return null;
			}
			
		});
		
		timed(registry, false, "one");
		timed(registry, true, "two");
		
		System.out.println(registry.scrape("application/openmetrics-text; version=1.0.0; charset=utf-8"));
	}
	
	static void timed(PrometheusMeterRegistry registry, boolean histogram, String methodName) throws InterruptedException {
		Sample sample = Timer.start(registry);
		Thread.sleep(500);
		sample.stop(Timer.builder("example").publishPercentileHistogram(histogram).tag("method-name",  methodName).register(registry));
	}

}

Example with Spring involved: spring-projects/spring-boot#40907 (comment)

Expected behavior
No exception; using Timer with and without histogram should work.

Additional context
Originally reported here: spring-projects/spring-boot#40907

[jonatan-ivanov]

Thanks for the issue and thanks Andy for the reproducer!

Workaround

It seems this is only an issue with the Prometheus 1.x Client, using the 0.x Client does not seem to have this issue. As a workaround you can downgrade to the 0.x client by using micrometer-registry-prometheus-simpleclient instead of micrometer-registry-prometheus.
Please see the release notes of Micrometer 1.13.0 as well as the Migration Guide. In a nutshell: we upgraded micrometer-registry-prometheus from Prometheus Client 0.x to 1.x but also kept the old client as-is under a different artifact: micrometer-registry-prometheus-simpleclient.
(See a code example below.)

Investigation Notes

Here's another reproducer also proving that the 0.x Client does not have this issue (commented out):

io.micrometer.prometheusmetrics.PrometheusMeterRegistry registry = new io.micrometer.prometheusmetrics.PrometheusMeterRegistry(io.micrometer.prometheusmetrics.PrometheusConfig.DEFAULT);
// io.micrometer.prometheus.PrometheusMeterRegistry registry =  new io.micrometer.prometheus.PrometheusMeterRegistry(io.micrometer.prometheus.PrometheusConfig.DEFAULT);

Timer.builder("test").tag("index", "1").publishPercentileHistogram(false).register(registry).record(Duration.ofMillis(100));
Timer.builder("test").tag("index", "2").publishPercentileHistogram( true).register(registry).record(Duration.ofMillis(100));
System.out.println(registry.scrape());

Doing such thing seems quite weird to me, it produces an output like this:

# HELP test_seconds  
# TYPE test_seconds summary
test_seconds_count{index="1",} 1.0
test_seconds_sum{index="1",} 0.1
test_seconds_bucket{index="2",le="0.001",} 0.0
[...]
test_seconds_bucket{index="2",le="+Inf",} 1.0
test_seconds_count{index="2",} 1.0
test_seconds_sum{index="2",} 0.2

This means that test_seconds is neither a Prometheus Summary (it has buckets) nor a Prometheus Histogram (it does not have buckets). The TYPE says summary which is not true. I think this was possible since with the 0.x Client Micrometer had bigger freedom to do things (even if those things were invalid) and it was able to produce "invalid" results. The Prometheus 1.x Client is much restrictive so some things are not possible anymore (in general I think this is a good thing).

Let me reach out to the Prometheus Client maintainer but right now I feel the output should not be considered as valid and this should be fix in your code by

• either turning histograms on for every Timer/LongTaskTimer/DistributionSummary with the same name.
• or use a different name for meters where histograms are on/off

[knoobie]

Thanks for investigation; I've applied your mentioned workaround with adding histogram to all those cases.

Example / Context / Reasoning behind this: We use @Timed for all external methods of, for example, Controller to get information about througput - to group them all, all use e.g. controller_1 as name and add additional tags like "method=get" and so on to identify each method within the service. To reduce "costs" of storing operations where we know that they are fast / cheap we skip the histogram flag to reduce the amount of scraped data points.

[jonatan-ivanov]

If you are using a framework like Spring Boot, Quarkus, Micronaut, etc, those controllers are already instrumented so you don't need to re-instrument them with @Timed.

Btw, you can apply @Counted where you don't care about the duration and @Timed where you do (@Counted is also cheaper). In Prometheus, metrics under the same name should have the same type, the previous behavior was because Micrometer was able to generate basically any output which is not the case anymore.

[jonatan-ivanov]

After talking to the maintainer of the Prometheus Client, there is one more workaround you can do right now: in Prometheus the histogram is valid if it has a single bucket: +Inf. You can "approximate" this behavior with the current Micrometer API without any modifications by having two buckets using .serviceLevelObjectives(...):

PrometheusMeterRegistry registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
Timer.builder("test").tag("index", "1")
    .serviceLevelObjectives(Duration.ofNanos(1)) // or you can use any meaningful value, like 100 ms
    .register(registry)
    .record(Duration.ofMillis(100));
Timer.builder("test").tag("index", "2")
    .publishPercentileHistogram( )
    .register(registry)
    .record(Duration.ofMillis(100));

System.out.println(registry.scrape());

Doing this means that sometimes you will have all the buckets and sometimes you will have only two buckets (not sure if it is good enough in your case). Unfortunately @Timed can't do SLOs right now but we can consider adding it, we already have a PR: #5145 please let us know if this is something you would use (by commenting on it/adding a 👍🏼).

I think I still not recommend doing the above since the buckets are different and because of that they are not aggregatable which kind of defeats the purpose of the histogram and I would still go with having summary and histogram with different names but since you are already doing this, you probably already aware of the consequences.

With this, let me close this issue, please let us know if these don't help and we can reopen.

[opendevrunner]

Hello,

the issue exists if the 'publishHistogram()' can not be used. The same ClassCastException occur if a histogram with defined ranges is used:

Timer.builder("test").tag("index", "1")
	.serviceLevelObjectives(Duration.ofNanos(1)) // or you can use any meaningful value, like 100 ms
	.register(registry)
	.record(Duration.ofMillis(100));
Timer.builder("test").tag("index", "2")
	.publishPercentiles(0.5, 0.75, 0.9, 0.95, 0.99,0.999 )
	.register(registry)
	.record(Duration.ofMillis(500));

From my point of view this is a breaking change as Meter with different tags are a normal case to differentiate results in Grafana. Such workarounds does not help. A solution without ClassCastExceptions and at least error reporting of problematic meters is required. The current implementation does not report ANY metric back to Prometheus.

Please recheck the decision to close the issue.