Add JVM metrics for total heap memory usage from MemoryMXBean

[liustanley]

I opened open-telemetry/opentelemetry-java-instrumentation#9859 and was told to open an issue here in the specification in order to propose this change.

Is your feature request related to a problem? Please describe.

The heap memory JVM metrics are currently being obtained from MemoryPoolMXBean.

I'm having difficulty calculating the correct maximum total values from the current jvm.memory.* metrics because they're divided by the memory pool attribute. For example, if I want to calculate the maximum total heap memory usage, it's possible to sum up the values from:

max:process.runtime.jvm.memory.usage{type:heap, pool:eden_space} + max:process.runtime.jvm.memory.usage{type:heap, pool:tenured_gen} + max:process.runtime.jvm.memory.usage{type:heap, pool:survivor_space}

But if a service has multiple JVM instances, then it becomes difficult to distinguish between the JVM instances and this query will sum across all JVM instances in this service.

If there was another metric for process.runtime.jvm.total.memory.usage (from MemoryMXBean), then we could query max:process.runtime.jvm.total.memory.usage{type:heap} for the correct total maximum heap memory across all JVM instances in the same service.

Describe the solution you'd like

I'd like to add the following heap memory JVM metrics from MemoryMXBean:

• process.runtime.jvm.total.memory.usage
• process.runtime.jvm.total.memory.committed
• process.runtime.jvm.total.memory.init
• process.runtime.jvm.total.memory.limit

I can help open a PR for this if the feature request is approved.

Describe alternatives you've considered

No response

Additional context

No response

[jack-berg]

I'm not sure what metrics backend and query language you're using, but this seems to be a limitation of the tool.

If I'm using prometheus and want to query for the heap usage while preserving the dimension representing the instance, I would do something like:

sum by (instance)(process_runtime_jvm_memory_usage_bytes{type="heap"})

This filters for the series where type="heap", summing up the various individual pools, but preserves the instance dimension.

[liustanley]

@jack-berg The query you wrote is possible in the query language I'm using, but isolating the instance isn't possible in all of our use cases. For more context, we're making a universal dashboard for customers that should ideally work for any deployment out of the box, so we don't know what unique attributes can be used as a grouping key. We can expect customers to set some attributes set on the metrics to make sure they are distinct, but we don't know in advance what they would be.

If we had the total metrics from MemoryMXBean, then we could display accurate values for averages, maxes, mins, and sums across JVM instances/services even when the instance can't be isolated.

For example, we want to calculate the maximum total heap memory usage using the formula from above:

max:process.runtime.jvm.memory.usage{type:heap, pool:eden_space} + max:process.runtime.jvm.memory.usage{type:heap, pool:tenured_gen} + max:process.runtime.jvm.memory.usage{type:heap, pool:survivor_space}

But we can't isolate the instance in advance, so this formula could end up adding values from different JVM instances, which wouldn't represent the correct total maximum heap memory usage.

If we could query max:process.runtime.jvm.total.memory.usage{type:heap} instead, then this would display the correct value across JVM instances/services without requiring the customer to manually filter on their distinct attributes in order to get the correct value.

[jack-berg]

I don't understand why the tooling doesn't allow you to query for type:heap (i.e. ignore the pool attribute) on the existing process.runtime.jvm.memory.usage metric. Why does the presence of the additional pool attribute cause issues?

Spatial reaggregation is a fundamental part of the metrics data model. Both SDKs and the collector have tools to drop unwanted attributes (i.e. views for the SDK and the metrictransformprocessor for the collector). Even the versioning and stability requirements, which are particularly strict around metric attributes, allow adding new attributes to existing metrics that do not "break apart".

What you're asking for is essentially a special metric view with a reduced set of attributes, and would result in the same information being sent up in two different ways. Semantic conventions should not do this type of thing.

even when the instance can't be isolated.
But we can't isolate the instance in advance, so this formula could end up adding values from different JVM instances,

Maybe this is the crux of your issue. Why can't the instance be isolated? Are you trying to provide a dashboard which works when regardless of service.instance.id is present or not? If so, consider making service.instance.id a requirement for this feature. Many aspects of JVM runtime conventions break down if there isn't an attribute like service.instance.id which uniquely identifies the instance within a particular service. When service.instance.id is not present, the single write principle is being violated, since two unique JVMs can be emitting time series with the same identity.

[liustanley]

@jack-berg The presence of the additional pool attribute doesn't cause issues, but the metric values are divided into each memory pool attribute which results in a different total metric value. For example:

From MemoryMXBean, process.runtime.jvm.memory.usage{type:heap} has a data point with value 100.

From MemoryPoolMXBean, This same metric would be reported as:

• process.runtime.jvm.memory.usage{type:heap, pool:eden_space} data point value = 50
• process.runtime.jvm.memory.usage{type:heap, pool:tenured_gen} data point value = 25
• process.runtime.jvm.memory.usage{type:heap, pool:survivor_space} data point value = 25

This causes issues because we then need to aggregate the pool values somewhere in our pipeline. I agree that it would be ideal if the instance id could be guaranteed, but we can't guarantee this since it's intended for a universal dashboard that should work with many different kinds of deployments.

The metricstransformprocessor seemed like it would work for aggregating the pool values, but then we realized that it wouldn't work for Gateway deployments because it would require all of the pool values to be sent to the same Collector instance. This could be done with the loadbalancingexporter, but it would add a lot of complexity and metrics are still in development.

Would it be possible to add the total value from MemoryMXBean as an attribute on the existing metrics for usage, committed, init, and limit if adding new metrics isn't possible?

[jack-berg]

This causes issues because we then need to aggregate the pool values somewhere in our pipeline. I agree that it would be ideal if the instance id could be guaranteed, but we can't guarantee this since it's intended for a universal dashboard that should work with many different kinds of deployments.

You can reject the data or disable the dashboard if service.instance.id isn't included. I think doing so is reasonable because without it, there's a high probability the single writer principle is being violated. See #311 for an issue attempting to make service.instance.id always present.

Would it be possible to add the total value from MemoryMXBean as an attribute on the existing metrics for usage, committed, init, and limit if adding new metrics isn't possible?

No, the memory usage is a high cardinality value and metric attributes need to be limited to low cardinality values. Also, attributes are for describing the data, not for encoding values.

I don't think this is an issue with JVM memory metrics or JVM metrics in particular, but actually a broader question of ensuring there is always way to do spatial reaggregation in a backend, which requires service.instance.id. I would recommend @open-telemetry/specs-semconv-maintainers close this particular issue but reference the use case on #311.

[joaopgrassi]

Closing this as no activity and I agree with the comments above from @jack-berg. @liustanley feel free to follow #311 or post your concerns there as well.